Improving Alignment Tools for Enhanced, More Accessible Assessments

Size: px

Start display at page:

Download "Improving Alignment Tools for Enhanced, More Accessible Assessments"

Linette Holland
6 years ago
Views:

1 Improving Alignment Tools for Enhanced, More Accessible Assessments Goals. Produce an electronic, CD-based alignment analysis process.. Adapt process to be more applicable to the full range of expectations and assessments of all students with disabilities.. Make procedures applicable for everygrade assessments and vertical scaling.

2 Deliverables. A CD of the alignment analysis process with adaptations for students with disabilities and vertical alignment. Written report on field-testing the CD alignment analysis process. Written report on student with disabilities procedural adaptations 4. Written report on vertical alignment expansions to the process 5. Workshop for training state and district staff to use CD-based alignment analysis process

3 Work Plan. Management team provides general guidance. Three TILSA working groups work with consultants. Working groups identify issues, questions, and specifications 4. Pilot test procedures 5. Analyze results and revise procedures 6. Beta test procedures 7. Analyze results and finalize procedures 8. Develop documentation and write reports 9. Conduct dissemination workshop

4 Benefits CD-based Alignment Analysis Process. Improved training of reviewers. Easier search for standards to match items. Greater access to people 4. Quicker turn around of analysis 5. Reviewers can be at different locations 6. Reduced errors in entry 7. More reports

5 Alignment Analysis Process Main Criteria and Judgments Categorical Concurrence Depth-of-Knowledge Consistency Range-of-Knowledge Correspondence Balance of Representation Source of Challenge (item quality) Specify and Enter Standards Train Reviewers (6 to 8) Use Consensus Process to Define Objectives DOK Code DOK and Objectives for Each Assessment Task Analyze Reviewers Coding Report Findings

6 Structure of the Automated Alignment Process Registration Group Leader Reviewers Standards/Goals/Objectives Entry Process Training on Depth-of-Knowledge Levels Phase I Consensus Process on Assigning DOK Levels to Objectives Phase II Coding of Assessment Tasks Phase III Analysis of Coding Phase IV Reporting

7 Objectives Coding Sheet End of Instruction High School US History Reviewer: DOK. Analyze causes, key events, and the effects of the Civil War and Reconstruction. A. Examine the economic and philosophical differences between the North and South, as exemplified by such persons as Daniel Webster and John C. Calhoun. B. Trace the events leading to secession and war. C. Identify leaders on both sides of the war (e.g., Abraham Lincoln, Ulysses S. Grant, Jefferson Davis, Robert E. Lee, Frederick Douglass, and William Lloyd Garrison). D. Interpret the importance of critical developments in the war, such as major battles, the Emancipation Proclamation, and Lee s surrender at Appomattox. E. Describe life on the battlefield and on the home front. F. Relate the basic provisions and postwar impact of the th, 4 th, and 5 th Amendments to the United States Constitution. G. Recognize the impact of Reconstruction policies on the South, and Southern reaction during the Reconstruction era (e.g., Black Codes, Ku Klux Klan, and Jim Crow laws).. Analyze the impact of immigration on American Society. A. Detail the contributions of various immigrant, cultural, and ethnic groups (e.g., Irish, Chinese, Italians, and Scandinavians).

8 Coding Form State Reviewer Date: Grade Test Form Item Number Item DOK Primary Obj Secondary Obj Secondary Obj Source of Challenge Notes

34 . Summary tables as included in the reports Table Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Title Standards Goals # Objs # Level by Objective Level # of objs by Level % w/in std by Level Mean Hits S.D. Categorical Concurr. Acceptable I. Reading Process YES II. Responding to Text YES III. Information and Research NO Total Includes one generic goal because coded items did not correspond to existing goals.

35 Table Depth-of-Knowledge Consistency Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Standards Level by Objective Hits Level of Item w.r.t. Standard % Under % At % Above Depth-of- Knowledge Consistency Title Goals # Objs # Level # of objs by Level % w/in std by Level M S.D. M S.D. M S.D. M S.D. I. Reading Process YES II. Responding to Text NO III. Information and Research NO Total Includes one generic goal because coded items did not correspond to existing goals.

36 Table Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Standards Level by Objective Level =Recall Level 4=Complex Reasoning Hits Range of Objectives # Objs Hit % of Total Range of Know. Accept. Balance Index ( perfect 0 no balance) % Hits in Std/Ttl Hits Index Balance of Representation Acceptable Title I. Reading Process Goals # Objs # 8. Level # of objs by Level 6 % w/in std by Level 67 M.50 S.D M. S.D..0 M 4 S.D. NO M 8 S.D. 7 M.68 S.D.. YES II. Responding to Text YES WEAK III. Information and Research NO.0.65 NO Total Includes one generic goal because coded items did not correspond to existing goals.

37 Table 4 Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Alignment Criteria Standards Categorical Concurrence Depth-of- Knowledge Consistency Range of Knowledge Balance of Representation I. Reading Process YES YES NO YES II. Responding to Text YES NO YES WEAK III. Information and Research NO NO NO NO

38 . Item by Rater Table showing DOK for the item and objective State E Math Grade 4 Item codes by Reviewers Item # Rater A Rater A Rater A Rater A4 DOK Objective DOK Objective DOK Objective DOK Objective A A A A C B C C A A A A 4 A A A A 5 A8 A8 A8 A8 6 D D D D 7 A A A A 8 A A A A 9 A A A A 0 C C C C A A A A CGEN CGEN CGEN CGEN

39 . Items coded by each rater for each standard. State E Math Rater Rater Rater Summary Rater Obj A A A Four or more Raters Three Raters Two Raters One Rater A4 A I I I A A A4 I7 I7 I7, I8 I7 I8 A5 I5, I7 I5, I7 A6 I8 I8

40 4. Intraclass Correlation Report Table B-9 Reliability of Depth-of-Knowledge Levels Ratings of Items for State E in Science Grade Number of Reviewers Number of Items Alpha* 95% CI Lower- Upper State E Science Chemistry External *Average Measure Intraclass Correlation

Course Description: Technology:

Course Description: Technology: Cambridge AICE History I Mr. Trotter james.trotter@mnps.org John Overton High School Class Website: www.trotteraice.wordpress.com Course Description: AICE* History I is an in-depth study of US History