Improving Alignment Tools for Enhanced, More Accessible Assessments Goals. Produce an electronic, CD-based alignment analysis process.. Adapt process to be more applicable to the full range of expectations and assessments of all students with disabilities.. Make procedures applicable for everygrade assessments and vertical scaling.
Deliverables. A CD of the alignment analysis process with adaptations for students with disabilities and vertical alignment. Written report on field-testing the CD alignment analysis process. Written report on student with disabilities procedural adaptations 4. Written report on vertical alignment expansions to the process 5. Workshop for training state and district staff to use CD-based alignment analysis process
Work Plan. Management team provides general guidance. Three TILSA working groups work with consultants. Working groups identify issues, questions, and specifications 4. Pilot test procedures 5. Analyze results and revise procedures 6. Beta test procedures 7. Analyze results and finalize procedures 8. Develop documentation and write reports 9. Conduct dissemination workshop
Benefits CD-based Alignment Analysis Process. Improved training of reviewers. Easier search for standards to match items. Greater access to people 4. Quicker turn around of analysis 5. Reviewers can be at different locations 6. Reduced errors in entry 7. More reports
Alignment Analysis Process Main Criteria and Judgments Categorical Concurrence Depth-of-Knowledge Consistency Range-of-Knowledge Correspondence Balance of Representation Source of Challenge (item quality) Specify and Enter Standards Train Reviewers (6 to 8) Use Consensus Process to Define Objectives DOK Code DOK and Objectives for Each Assessment Task Analyze Reviewers Coding Report Findings
Structure of the Automated Alignment Process Registration Group Leader Reviewers Standards/Goals/Objectives Entry Process Training on Depth-of-Knowledge Levels Phase I Consensus Process on Assigning DOK Levels to Objectives Phase II Coding of Assessment Tasks Phase III Analysis of Coding Phase IV Reporting
Objectives Coding Sheet End of Instruction High School US History Reviewer: DOK. Analyze causes, key events, and the effects of the Civil War and Reconstruction. A. Examine the economic and philosophical differences between the North and South, as exemplified by such persons as Daniel Webster and John C. Calhoun. B. Trace the events leading to secession and war. C. Identify leaders on both sides of the war (e.g., Abraham Lincoln, Ulysses S. Grant, Jefferson Davis, Robert E. Lee, Frederick Douglass, and William Lloyd Garrison). D. Interpret the importance of critical developments in the war, such as major battles, the Emancipation Proclamation, and Lee s surrender at Appomattox. E. Describe life on the battlefield and on the home front. F. Relate the basic provisions and postwar impact of the th, 4 th, and 5 th Amendments to the United States Constitution. G. Recognize the impact of Reconstruction policies on the South, and Southern reaction during the Reconstruction era (e.g., Black Codes, Ku Klux Klan, and Jim Crow laws).. Analyze the impact of immigration on American Society. A. Detail the contributions of various immigrant, cultural, and ethnic groups (e.g., Irish, Chinese, Italians, and Scandinavians).
Coding Form State Reviewer Date: Grade Test Form Item Number Item DOK Primary Obj Secondary Obj Secondary Obj Source of Challenge Notes 4 5 6 7
. Summary tables as included in the reports Table Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Title Standards Goals # Objs # Level by Objective Level # of objs by Level % w/in std by Level Mean Hits S.D. Categorical Concurr. Acceptable I. Reading Process 8. 6 67.50 6.60 YES II. Responding to Text 9 5 75 45.50.4 YES III. Information and Research 4 5 75.67. NO Total 5 4. 0 8 40 5 6. 7.55 Includes one generic goal because coded items did not correspond to existing goals.
Table Depth-of-Knowledge Consistency Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Standards Level by Objective Hits Level of Item w.r.t. Standard % Under % At % Above Depth-of- Knowledge Consistency Title Goals # Objs # Level # of objs by Level % w/in std by Level M S.D. M S.D. M S.D. M S.D. I. Reading Process 8. 6 67.50 6.60 6 64 7 4 YES II. Responding to Text 9 5 75 45.50.4 76 46 NO III. Information and Research 4 5 75.67. 64 8 6 8 0 0 NO Total 5 4. 0 8 40 5 6. 7.55 65 44 8 Includes one generic goal because coded items did not correspond to existing goals.
Table Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Standards Level by Objective Level =Recall Level 4=Complex Reasoning Hits Range of Objectives # Objs Hit % of Total Range of Know. Accept. Balance Index ( perfect 0 no balance) % Hits in Std/Ttl Hits Index Balance of Representation Acceptable Title I. Reading Process Goals # Objs # 8. Level # of objs by Level 6 % w/in std by Level 67 M.50 S.D. 6.60 M. S.D..0 M 4 S.D. NO M 8 S.D. 7 M.68 S.D.. YES II. Responding to Text 9 5 75 45.50.4 9.67.5 8 4 YES 74 9.6.06 WEAK III. Information and Research 4 5 75.67..67. 4 0 NO.0.65 NO Total 5 4. 0 8 40 5 6. 7.55 4.89.66 54 6.49.4 Includes one generic goal because coded items did not correspond to existing goals.
Table 4 Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria State F Grade 5 Language Arts (Number of Assessment Items 40 Multiple Choice Items) Alignment Criteria Standards Categorical Concurrence Depth-of- Knowledge Consistency Range of Knowledge Balance of Representation I. Reading Process YES YES NO YES II. Responding to Text YES NO YES WEAK III. Information and Research NO NO NO NO
. Item by Rater Table showing DOK for the item and objective State E Math Grade 4 Item codes by Reviewers Item # Rater A Rater A Rater A Rater A4 DOK Objective DOK Objective DOK Objective DOK Objective A A A A C B C C A A A A 4 A A A A 5 A8 A8 A8 A8 6 D D D D 7 A A A A 8 A A A A 9 A A A A 0 C C C C A A A A CGEN CGEN CGEN CGEN
. Items coded by each rater for each standard. State E Math Rater Rater Rater Summary Rater Obj A A A Four or more Raters Three Raters Two Raters One Rater A4 A I I I A A A4 I7 I7 I7, I8 I7 I8 A5 I5, I7 I5, I7 A6 I8 I8
4. Intraclass Correlation Report Table B-9 Reliability of Depth-of-Knowledge Levels Ratings of Items for State E in Science Grade Number of Reviewers Number of Items Alpha* 95% CI Lower- Upper State E Science Chemistry External 5 60.8.7-.88 *Average Measure Intraclass Correlation