Overview of Assessment in Flagler County September 2016
Legislation Impacting Assessment First iteration of 1008.22 (6) F.S. 1008.22(6) LOCAL ASSESSMENTS. (a) Measurement of student performance in all subjects and grade levels, except those subjects and grade levels measured under the statewide, standardized assessment program described in this section, is the responsibility of the school districts. (b) Except for those subjects and grade levels measured under the statewide, standardized assessment program, beginning with the 2014-2015 school year, each school district shall administer for each course offered in the district a local assessment that measures student mastery of course content at the necessary level of rigor for the course. As adopted pursuant to State Board of Education rule, course content is set forth in the state standards required by s. 1003.41 and in the course description. Local assessments may include: 1. 2. 3. 4. 5. Statewide assessments. Other standardized assessments, including nationally recognized standardized assessments. Industry certification assessments. District-developed or district-selected end-of-course assessments. Teacher-selected or principal-selected assessments. (c) Each district school board must adopt policies for selection, development, administration, and scoring of local assessments and for collection of assessment results. Local assessments implemented under subparagraphs (b)4. and 5. may include a variety of assessment formats, including, but not limited to, project-based assessments, adjudicated performances, and practical application assignments. For all English Language Arts, mathematics, science, and social studies courses offered in the district that are used to meet graduation requirements under s. 1002.3105, s. 1003.4281, or s. 1003.4282 and that are not otherwise assessed by statewide, standardized assessments, the district school board must select the assessments described in subparagraphs (b)1.-4.
School Board Policy 415 I. No student shall be excluded from participation in, be denied the benefit of, or be subjected to discrimination under any District testing progrm on the basis of race, national origin, marital status, handicap, or religion. Test accommodations consistent with tate and federal requirements shall be made for students with disabilities, students with 504 plans, and ELL students. II. The parent, or guardian, as defined by Florida Statutes, of each student must be notified of the progress of the student towards achieving state and district expectations and proficiency in reading, science, writing and mathematics. A student's state assessment results must be reported to the parent or guardian. III. It is the policy of the Flagler County School District to adopt local assessments for the purpose of measuring student performance in all subjects and grade levels not measured under the statewide, standardized assessment program. Beginning with the 2014-15 school year, Flagler County shall administer for each course offered in the district, a state, national, or local student assessment that measures mastery of the course content, as described in the state-adopted course descriptions. These assessments may include statewide assessments, other standardized assessments, including nationally recognized standardized assessments, industry certification examinations, and/or district developed/selected end of course assessments. In the event that none of the aforementioned examinations exist, principal approved end of course assessments will be utilized to measure course mastery. IV. Development of Local Assessments: Beginning with the 2014-15 school year, each school's course selection sheet will be used to identify courses for which a "local" assessment is required. Teachers of these courses will have input into content focus reports (test designs) which includes the format of the test, standards to be assessed, and complexity of items. Only courses for which standards exist will have assessments developed. Content focus reports (test designs) may include performance or project-based tasks. Assessments items developed by the Florida Item Bank and Test Platform (IBTP), the Central Florida Assessment Collaborative (CFAC), and other consortia will be used to create assessments that mirror content focus reports (test design). In the event that items do not exist in Florida IBTP, CFAC, or other consortia banks, a teacher shall be required to develop items for the test banks being used. Results of assessments will be collected by each school. Security of items shall be maintained by the school and teacher for all assessments mentioned above. All teachers will adhere to the Flagler Schools Assessment Security Agreement.
Legislation Impacting Assessment Second iteration of 1008.22 (6) F.S. 1008.22(6) LOCAL ASSESSMENTS. (6) LOCAL ASSESSMENT OF STUDENT PERFORMANCE ON STATE STANDARDS. (a) Measurement of student performance is the responsibility of school districts except in those subjects and grade levels measured under the statewide, standardized assessment program described in this section. When available, instructional personnel must be provided with information on student achievement of standards and benchmarks in order to improve instruction. (b) The Commissioner of Education shall assist and support districts in measuring student performance on the state standards by maintaining a statewide item bank, facilitating the sharing of developed tests or test items among school districts, and providing technical assistance in best assessment practices. The commissioner may discontinue the item bank if he or she determines that district participation is insufficient for its sustainability. Available online at http://www.leg.state.fl.us/statutes/?app_mode=display_statute&url=1000-1099/1008/sections/1008.22.html
Teacher Evaluation Statute That Is Linked to Assessments & Use of Student Performance Data 1012.34 (3)(a)(1) F.S.: Evaluation Procedures and Criteria Performance of students. At least one-third of a performance evaluation must be based upon data and indicators of student performance in accordance with subsection (7). This portion of the evaluation must include growth or achievement data of the teacher s students or, for a school administrator, the students attending the school over the course of at least 3 years. If less than 3 years of data are available, the years for which data are available must be used. The proportion of growth or achievement data may be determined by instructional assignment.
Teacher Evaluation Statute That Is Linked to Assessments & Use of Student Performance Data (7) MEASUREMENT 1012.34 (7) OF STUDENT PERFORMANCE. 1 (a) The Commissioner of Education shall approve a formula to measure individual student learning growth on the statewide, standardized assessments in English Language Arts and mathematics administered under s. 1008.22. The formula must take into consideration each student s prior academic performance. The formula must not set different expectations for student learning growth based upon a student s gender, race, ethnicity, or socioeconomic status. In the development of the formula, the commissioner shall consider other factors such as a student s attendance record, disability status, or status as an English language learner. The commissioner may select additional formulas to measure student performance as appropriate for the remainder of the statewide, standardized assessments included under s. 1008.22 and continue to select formulas as new assessments are implemented in the state system. After the commissioner approves the formula to measure individual student learning growth, the State Board of Education shall adopt these formulas in rule. (b) Each school district shall measure student learning growth using the formulas approved by the commissioner under paragraph (a) and the standards for performance levels adopted by the state board under subsection (8) for courses associated with the statewide, standardized assessments administered under s. 1008.22 no later than the school year immediately following the year the formula is approved by the commissioner. We know this as the VAM statute that impacts grades 4-10 ELA, grades 4-8 Mathematics, and grades 8-9 Algebra 1. For grades and subjects not assessed by statewide, standardized assessments, each school district shall measure student performance using a methodology determined by the district. Flagler has an Evaluation Committee that has determined a set of agreements and bylaws for how to use student performance in teacher evaluations for district measured areas.
How The Evaluation System Links to Assessment (Requires VAM/SPI to be measurable) VAM arrives in a 1.0-4.0 score (Score is weighted by 66 to provide a max score of 264) Our Evaluation Committee decided to do the same with Non-VAM/SPI measures. (1.0-4.0 score) 496 points: Principal Evaluation (62%) 40 points: IPLP (5%) 264 points VAM/SPI (33%) 800 Total Points
Language Pertaining to Statewide Assessment Program 1008.22(4): Statewide Assessments SCHOOL PARTICIPATION IN THE STATEWIDE, STANDARDIZED ASSESSMENT PROGRAM. Each public school shall participate in the statewide, standardized assessment program in accordance with the assessment and reporting schedules and the minimum and recommended technology requirements published by the Commissioner of Education. District school boards shall not establish school calendars that conflict with or jeopardize implementation of the assessment program. All district school boards shall report assessment results using the state management information system. Performance data shall be analyzed and reported to parents, the community, and the state. Student performance data shall be used by districts in developing objectives for the school improvement plan, evaluating instructional personnel and administrative personnel, assigning staff, allocating resources, acquiring instructional materials and technology, implementing performance-based budgeting, and promoting and assigning students to educational programs. The analysis of student performance data must also identify strengths and needs in the educational program and trends over time. The analysis must be used in conjunction with the budgetary planning processes developed pursuant to s. 1008.385 and the development of remediation programs.
Law Pertaining to Percent of Time Tested 1008.22(7)(d): The 5% rule A school district may not schedule more than 5 percent of a student s total school hours in a school year to administer statewide, standardized assessments and district-required local assessments. The district must secure written consent from a student s parent before administering district-required local assessments that, after applicable statewide, standardized assessments are scheduled, exceed the 5 percent test administration limit for that student under this paragraph. The 5 percent test administration limit for a student under this paragraph may be exceeded as needed to provide test accommodations that are required by an IEP or are appropriate for an English language learner who is currently receiving services in a program operated in accordance with an approved English language learner district plan pursuant to s. 1003.56. Notwithstanding this paragraph, a student may choose within a school year to take an examination or assessment adopted by State Board of Education rule pursuant to this section and ss. 1007.27, 1008.30, and 1008.44.
5% Analysis 5% Once this statute came into effect, our district analyzed how much time was being spent on state mandated and/or district mandated assessments Went into effect April 14, 2015 Does not apply to student-elected common placement testing or student-elected accelerated testing (e.g., AP, IB, AICE, Industry Certifications) Does not apply to students requiring test accommodations as required by IEP or as appropriate for an ELL
Elementary School (Grades K - 3) District & State Statute Mandates at least 720 hours (43,200 minutes of instruction) at a minimum for Grades K-3. (All less than the 45 hours or 5% max) Grade FAIR 1/ FLKRS FAIR 2 FAIR 3 State Reading or SAT 10 I Ready State Math or SAT 10 State or District Science State or District Social Studies Total Minutes spent on state or district testing Percentage of Allocated Hours K 60 0 0 190 45 70 0 0 365 minutes (6.1 hrs) 1 0 0 0 150 45 120 0 0 315 minutes (5.25 hrs) 2 0 0 0 140 45 90 0 0 275 minutes (4.6 hrs) 3 45 45 45 160 0 160 0 0 455 minutes (7.58 hrs) 0.8% 0.73% 0.64% 1.05%
Elementary School (Grades 4-6) District & State Statute Mandates at least 900 hours (54,000 minutes of instruction) at a minimum for Grades 4-6. (All less than the 45 hours or 5% max) Electives are school-based now at the elementary level Grade FAIR 1 FAIR 2 FAIR 3 State Writing State Reading State Math State or District Science State or District Social Studies Electives Total Minutes spent on state or district testing Percentage of Allocated Hours 4 45 45 45 120 160 160 0 0 0 575 minutes (9.6 hrs) 5 45 45 45 120 160 160 160 45 0 735 minutes (12.25 hrs) 6 45 45 45 120 170 180 0 0 0 605 minutes (10.1 hrs) 1.1% 1.36% 1.12%
Middle School (Grades 7-8) District & State Statute Mandates at least 900 hours (54,000 minutes of instruction) at a minimum for Grades 7-8. (All less than the 45 hours or 5% max) Grade FAIR 1 FAIR 2 FAIR 3 State Writing State Reading State Math State or District Science State or District Social Studies Electives* Total Minutes spent on state or district testing Percentage of Allocated Hours 7 45 45 45 120 170 180 100 160 (Civics EOC) 300 1,165 minutes (19.42 hrs) 2.16% 8 45 45 45 120 170 180 160 (State) 100 300 1,165 minutes (19.42hrs) 2.16% *300 minutes estimated for grades 7-8 electives (3 electives at 100 minutes per district final)
High School (Grades 9-12) District & State Statute Mandates at least 900 hours (54,000 minutes of instruction) at a minimum for Grades 9-12. (All less than the 45 hours or 5% max) Grade FAIR 1 FAIR 2 FAIR 3 State Writing State Reading State Math State or District Science State or District Social Studies Electives* Total Minutes spent on state or district testing Percentage of Allocated Hours 9 45 45 45 120 170 180 (Alg 1) 160 (Bio) 90 270 1,135 minutes (18.92 hrs) 2.11% 10 45 45 45 120 170 180 (Geo) 90 90 270 1,065 minutes (17.75 hrs) 1.97% 11 0 0 0 0 0 180 (Alg 2) 90 160 (US History) 410 (360 electives + 50 PERT) 840 minutes (14 hrs) 1.6% 12 0 0 0 0 0 0 90 90 450 630 minutes (10.5 hrs) 1.17% *90 minutes per elective estimated for grades 9-12 electives (Estimate 3 electives for grades 9 & 10, 4 electives for grade 11, and 5 electives for grade 12)
Adding School-Based Testing to Analysis We created a spreadsheet and asked schools to add any school based routine assessments they give, the number of administrations as well as the estimated time in minutes per administration. This information was added to our state and district totals Name of School-Based Routine Reading or ELA Exams # of Administrations per year Estimated minutes of routine exams per school Total # of minutes of Routine Reading of ELA Exams Name of School-Based Routine Math Exams # of Administrations per year Estimated minutes of routine exams per school Total # of minutes of Routine Math Exams
Elementary School (Grades K - 3) District, State, & School Includes school-based final exams for special area teachers Includes bi-weekly and quarterly assessments for reading and math Grade Total Minutes Spent on State or District Testing Percentage of time on State or District Testing Additional Minutes for Routine School- Based Assessments Total Minutes Spent on State, District, or School-Based Testing Percentage of Allocated Hours when Including School-Based Testing K 365 0.8% 1,620 minutes 1,985 minutes (33 hours) 1 315 0.73% 1,620 minutes 1,935 minutes (32.25 hours) 2 275 0.64% 1,620 minutes 1,895 minutes (31.6 hours) 3 455 1.05% 1,770 minutes 2,225 minutes (37.1 hours) 4.6% 4.5% 4.4% 5.2%
Elementary School (Grades 4-6) District, State, & School Includes school-based final exams for special area teachers Includes bi-weekly and quarterly assessments for reading and math Grade Total Minutes Spent on State or District Testing Percentage of time on State or District Testing Additional Minutes for Routine School- Based Assessments Total Minutes Spent on State, District, or School-BAsed Testing Percentage of Allocated Hours when Including School-Based Testing 4 575 (9.6 hours) 5 735 (12.25 hours) 6 605 (10.1 hours 1.1% 1,845 minutes 2,420 minutes (40.3 hours) 1.36% 1,845 minutes 2,580 minutes (43 hours) 1.12% 1,845 minutes 2,450 minutes (40.8 hours) 4.5% 4.8% 4.5%
Middle School (Grades 7-8) District, State, & School Electives already included in district-mandated assessments estimate Includes routine and school-based quarterly assessments for reading, math, science, and social studies Grade Total Minutes Spent on State or District Testing Percentage of time on State or District Testing Additional Minutes for Routine School- Based Assessments Total Minutes Spent on State, District, or School-BAsed Testing Percentage of Allocated Hours when Including School-Based Testing 7 1,165 minutes (19.42 hours) 8 1,165 minutes (19.42 hours) 2.16% 540 minutes 1,705 minutes (28.4 hours) 2.16% 540 minutes 1,705 minutes (28.4 hours) 3.16% 3.16%
High School (Grades 9-12) District, State, & School CDFs for electives are already calculated in for district-mandated assessments prior to routine school-based assessments Grade Total Minutes Spent on State or District Testing Percentage of time on State or District Testing Additional Minutes for Routine School- Based Assessments Total Minutes Spent on State, District, or School-Based Testing Percentage of Allocated Hours when Including School-Based Testing 9 1,135 minutes 2.11% 321 minutes 1,456 minutes (24.3 hours) 10 1,065 minutes 1.97% 214 minutes 1,279 minutes (21.3 hours) 11 840 minutes 1.6% 214 minutes 1,054 minutes (17.6 hours) 2.7% 2.37% 1.95% 12 630 minutes 1.17% 0 minutes 630 minutes (10.5 hours) 1.17%
School Board Policy 145 IV. Development of Local Assessments: Beginning with the 2014-15 school year, each school's course selection sheet will be used to identify courses for which a "local" assessment is required. Teachers of these courses will have input into content focus reports (test designs) which includes the format of the test, standards to be assessed, and complexity of items. Only courses for which standards exist will have assessments developed. Content focus reports (test designs) may include performance or project-based tasks. Assessments items developed by the Florida Item Bank and Test Platform (IBTP), the Central Florida Assessment Collaborative (CFAC), and other consortia will be used to create assessments that mirror content focus reports (test design). In the event that items do not exist in Florida IBTP, CFAC, or other consortia banks, a teacher shall be required to develop items for the test banks being used. Results of assessments will be collected by each school. Security of items shall be maintained by the school and teacher for all assessments mentioned above. All teachers will adhere to the Flagler Schools Assessment Security Agreement.
Initial Process The state created a statewide gap analysis master assessment plan. As of 2011, there were 2,882 courses offered statewide in k-12 636 courses had planned or existing assessments State tests, FL interim assessments, districtdeveloped hard to measure assessments, AP, IB, AICE, SAT, ACT, PERT 2,246 gap courses
Identification of courses needing Common District Finals (Gap Courses) 1.Examination of course selection sheets 2.Identified all courses with state/national assessments available 3.Reviewed remaining courses to see which courses had items available through a state-created item bank.
Flagler Courses At the high school level alone, there were... 199 Total courses MHS & FPC - 69 MHS - 67 FPC - 63
Item Writing Process 1. Identified courses for which a consortium had plans to write items and courses where there were no plans. 2. Examined our list of courses offered and determined which ones had a consortium writing items for it a. For courses that had items being developed by a consortium, we had teachers apply to be a part of it b. For any course that did not have items being developed, we had principals identify teachers to attend in-house item-writing training
Courses Assessed by State/National Exams EOC Courses: U.S. History/U.S. History Honors Intensive Reading English 1/English 1 Honors English 2/English 2 Honors Algebra 1/Algebra 1B/Algebra 1 Honors Geometry/Geometry Honors Algebra 2/Algebra 2 Honors Biology/Biology Honors Additional Courses: AP courses IB Courses Dual Enrollment Courses Math for College Readiness- PERT English for College Readiness- PERT CTE courses - Industry Certification Exams unless not available
Courses Using Items From Different Consortiums TV Production 1 TV Production 2 TV Production 3 TV Production 4 Intensive Science BioTech Integrated Science 1 Integrated Science 2 Marine Science 1 Marine Science 2 Anatomy & Physiology Genetics Physical Science Leadership Techniques Leadership Education 1 Leadership Skills Advanced Leadership Education 1 Advanced Leadership Education 2 Advanced Leadership Education 3 Advanced Leadership Education 4 Parenting 1 Parenting 2 Foundations of Web Design 1 Journalism 1 Journalism 2 Journalism 3 Journalism 4 Digital Design 1 Digital Design 2 Debate 1 Debate 2 Culinary Arts 1 Culinary Arts 2 Pre-Calculus Informal Geometry Trigonometry Statistics
Courses Using Items From Different Consortiums Personal Fitness Fitness Issues Weight Training 1 Weight Training 2 Weight Training 3 Comprehensive Fitness Power Weight Training 1 Individual/Dual Sports 1 Individual/Dual Sports 2 Individual/Dual Sports 3 Recreational Activities Outdoor Education Team Sports 1 Team Sports 2 Fitness Lifestyle Design Care & Prevention of Athletic Injuries Dance Technique I Dance Technique II Acting I Acting II Acting III Acting IV Technical Theater Design Production 1 Chorus 1 Chorus 2 Chorus 3 Chorus 4 Chorus 5 Chorus 6 Vocal Ensemble I Vocal Ensemble II Guitar 1 Band 1 Band 2 Band 3 Band 4 Band 5 Band 6
Courses Using Items From Different Consortiums Jazz Band 1 Jazz Band 2 Jazz Band 3 Jazz Band 4 Eurythmics 1 Eurythmics 2 Ceramics & Pottery 1 Ceramics & Pottery 2 Drawing 1 Drawing 2 Creative Photo 1 Creative Photo 2 French 1 French 2 Spanish 1/Honors Spanish 2/Honors/ Pre-IB Spanish 3/Honors/ Pre-IB Italian 1 Italian 2 American Sign Language 1 American Sign Language 2 Agricultural Foundations Vet Assist 1 Vet Assist 2 Vet Assist 3 Vet Assist 4 Early Childhood Education 1 Culinary Arts 1 Culinary Arts 2 Teacher Assistant 1-4 English 3 English 4/Honors Social Media Earth Space Science Chemistry/Honors Physics/Honors Economics with Financial Literacy/Honors American Government World History
Item Bank Test Platform Comprised of items for stateidentified gap courses Created by consortiums across the state Available for creation and admission of tests
Courses Requiring Item Writing Training Grooming and Salon Services Peer Counseling 1 Peer Counseling 2 Plant BioTech Agricultural BioTech II Agricultural BioTech III Landscaping and Turf Science IV Landscaping and Turf Science V Accounting Applications Business and Entrepreneurial Principles DCT OJT Guided Work Learning Diversified Career Technology I Diversified Career Technology II Technology Studies Power & Energy Tech I Power & Energy Tech II Power & Energy Tech III Electricity I Building Construction Tech I Building Construction Tech II Building Construction Tech III Culinary Arts 3 Culinary Arts 4 Financial Internship Marketing Essentials Marketing Applications Marketing Management
Courses Requiring Item Writing Training Sports, Rec, & Entertainment Applications Sports, Rec, & Entertainment Marketing Management User Interface Design II Web Scripting Fundamentals III IT Essentials Tech Support Services I Programming I SGA III SGA IV Safety & Driver s Ed Drawing III Painting I Painting II Commercial Photo I Commercial Photo II Commercial Photo III Arts Appreciation/ Humanities Dance Technique III Choreography Dance Rep I Dance Rep II Dance Rep III Technical Theater Design Production II Steel Drum I Steel Drum II Steel Drum III Steel Drum IV Music Technology & Sound Engineering I Music Technology & Sound Engineering II African American History
Courses Requiring Item Writing Training Learning Strategies Gifted Studies Fundamental English II Fundamental English IV Fundamental Explorations in Math 1 Health & Safety 9-12 Fundamental U.S. History Fundamental U.S. Government Social & Personal Skills Career Preparation Career Experience Fundamental English I Fundamental English III Fundamental Algebraic Skills Social Studies 9-12 Fundamental World History Career Education 9-12
Process for Creating Flagler CDFs For all non-state/national exams, identified teachers from each school to create a CFR For courses where items existed, district team created exams using teacher-created CFR For courses in which items did not exist, pulled in teachers to write items to match CFR
Content Focus Report (CFR) Development Teachers were provided with the following template and were asked to identify standards that could be assessed through a multiple choice test. They also identified: The number of questions per selected standard The DOK for each standard Item limitations if warranted Teachers were directed to the item specifications documents for their course, when available. Sample item specifications document from DOE for Teacher Assistant 1
Content Focus Report (CFR) Development Teacher were given the opportunity to include a performance task if the course warranted such. Worth up to 80% for a junior or senior level course in a series Worth up to 70% for a middle school, freshman, sophomore level course, or a course not in a series.
Item Writing Training for Identified Flagler Courses Teachers of identified gap courses were brought in for a 3 day training to include: The anatomy of an item Direction Line Stem Distractors Correct answer Review of Cognitive Complexity Item Conventions Item Bias Copyright Issues Item vetting
Item Review Process 01 Conventions Grammar, spelling, punctuation Items identified through the review process as not meeting this criteria were edited. 02 03 04 Sensitivity Bias Avoidance of sensitive topics Bias Avoid stereotypes and socioeconomic bias Copyright Source information For items that had contained cluing were sent back to the original author for revision. Item reviewers included people who received item writing training from the state and who acted as reviewers for state item banks
CDF Spreadsheet Information Included Date exam was needed by Schools offering that course Subject Area Course # and name Exam to meet 1008.22 Whether or not the exam would count as the final and the percent it would count for CFR/Exam Information location Course description link Organization who created the test items Item specifications link Comments Issues identified during the test.
52 223 Exams and Content Focus Reports created 168 Study Guides created CFR/Study Guide Page Teachers involved in item writing for courses 147 Signed up for CDF reviews. 67 in 2016, 80 in 2015
CDF Review Process Teachers reviewed the exam and provided feedback Suggested edits were reviewed and made when deemed appropriate Updated CFR and Study Guide provided on the district site
CDF Review What you may bring: A print out of your standards The Content Focus Report (HS courses may also bring the Study Guide) for your course, which can be found at http://flaglerschools.com/district/content-focus-reports Something to write with Headphones if your exam has audio items Process: Teachers will review the test items to determine if changes need to be made If changes need to be made, they will follow the guidelines provided below. If a teacher would like, they may review the district data available for the questions on the test. If changes need to be made, they will follow the guidelines provided below. Teachers will review the CFR and make suggested adjustments as needed to match any suggested modifications made to the test. Teachers will review the study guide, if available, and make suggested adjustments as needed.
CDF Review Guidelines for Improving Items: Whenever possible, please try to improve the current item. If you feel that an item could be improved, please: Provide the rationale for the change Example: The question is not aligned to the standard because... Non-example: I don t teach this. Indicate the change to be made. Please be explicit in the change and provide the language that should be added/removed/modified. If the question is okay, but the answer choices need improvement, please indicate the changes to be made to the answer choice. If you feel an item is not salvageable, please: Provide the rationale for removing the item Provide an alternate question with answer choices. If you choose to provide an alternate question, please see the Curriculum Specialist who is overseeing the CDF Review for the expectations for new items. If an alternate question is not provided and you wish to keep the same number of questions for that standard, a new item will be added if one is available. If one is not available, the CFR will be adjusted to reflect a reduction in items for that standard. Please note that every effort will be made to improve each item, but if the recommendation for improvement is not clear, adjustments will not be made at that time.
Standard Setting Process
Why are we standard setting? We are trying to improve our local processes We are trying to mirror state processes
Basics of Statistics Item discrimination P values Haladyna Charts All of this data is available in Performance Matters for these exams Our district also is consistently reviewing the Cronbach s alpha scores for each test which is a true statistical measure of test reliability.
Item Discrimination Item Discrimination: An indicator of how well the question can tell the difference between high and low performing students. An item that has an item discrimination below 0.2 should be looked at for the potential of bias, ambiguity, accuracy, validity, or other potential issues.
P Values (Item difficulty) A P Value refers to the degree of challenge for items based on the percentage of students who chose the correct answer. Thus, P values cannot be determined until AFTER a group of students have taken an exam or participated in an item. Our gut instinct is that an item with a p value of 0.5 (only 50% of students answered correctly) is a bad item. We need to rethink about our purpose of assessing. Should a test be full of items with P values all above 0.7? Test would be very easy and would have a very low item discrimination (items cannot differentiate between higher level learners and lower level learners) Should a test be full of items with P values all below 0.4?
Example of P values vs. Item Discrimination In the screenshot below, you can see that the item has an item discrimination below 0.2 (item doesn t differentiate between high level students and low level students) and has a P value of 0.91 meaning that 91% of students answered the question correctly. Logically, this also indicates a very low item discrimination because almost everyone is getting the question correct.
Haladyna Charts to View Item Responses (Easy items) Analyzing Easy Questions Identified using Haladyna Charts Correct responses begin to spike in green early in the graph compared to average score of student group
Haladyna Charts to View Item Responses (Average items) Analyzing Average Questions using Haladyna Charts Correct responses begin to spike in the middle of the graph compared to average score of student group
Haladyna Charts to View Item Responses (Challenging items) Analyzing Difficult Questions using Haladyna Correct responses begin to spike later in the graph compared to average score of student group In the example below, for students who earned a 90% on their exam, about 70% of those students answered this question correctly.
Haladyna Charts to View Item Responses (Ambiguous items or ineffective distractors) In the chart below, any item in red is an incorrect response. In the lightly shaded red, you can see that 80% of the students who earned an 80% chose a particular distractor that was incorrect meaning that even this group of students (earned an 80% so should be a high performing group) chose this particular distractor. Either a misconception wasn t addressed or the choice has ambiguity).
Example of State FSA EOC Leveling vs. Percentage of Points Earned
Flagler CDF Standard Setting & Leveling Level Raw Percentage Correct Range Percentage Equivalent To Be Entered into Gradebook 1 0-30% 15%-35% 2 31-44% 36%-69% 3 45%-59% 70%-85% 4 60%-79% 86%-95% 5 80%-100% 96%-100%
Flagler CDF Standard Setting & Leveling Mirrors the same logic that we use on state EOCs for each Level Applies to this round of CDFs, future CDFs, and any CDF s that were previously utilized at your schools for the 2015-16 school year (will require grade changes at your schools) For previous CDF s administered in 2015-16, this should only apply if the updated CDF grade using the conversion process would improve a student s grade Conversion Table was emailed to each school for use on CDFs (sample screenshot to the right) Red = Level 1 Yellow = Level 2 Green = Level 3 Blue = Level 4 Black = Level 5
Questions?