Every Student Succeeds Act Topic Discussion Guide Under the Every Student Succeeds Act (ESSA), Ohio will create a plan to better align our local, state and federal programs to help all students be successful. The Ohio Department of Education is committed to meaningfully engaging a diverse group of stakeholders through a variety of methods and opportunities to solicit thoughts, opinions and recommendations concerning provisions in Ohio s state plan. Everyone s input is required to create a plan that is deeply rooted in the needs of Ohio s students. Ohio is conducting a series of topic specific webinars. Each topic will have a detailed discussion guide. The first topic, Minimum N-size for subgroup evaluation, is discussed below. Minimum N-size for Subgroup Evaluation WHAT IS N-SIZE? The N-size is a statistical determination that is used for accountability and data reporting. Ohio will be reporting on the academic achievement and graduation rates of several groups of students that have historically not performed at the same levels as the rest of their peers. This is commonly referred to as the achievement gap. These groups include students with disabilities, children in poverty and several others. Schools and districts are held accountable for the performance of these students to ensure all students are learning. To do so, the state must determine how many students a school must have in each subgroup before the student subgroup is included in the analysis. This number needs to include a fair and valid number of students, and simultaneously protect student privacy. WHAT DOES ESSA REQUIRE? States must identify an N-size, or the minimum number of students from a group that a school or district would need for that group to count as a viable group for evaluation purposes in the accountability system. This determination must be made with input from Ohio stakeholders. This determination will be used for disaggregated reporting and accountability for subgroups on academic performance in mathematics and English language arts, graduation and participation in state assessments. New subgroups have been added for reporting purposes (military dependents, homeless, migrant, foster children). The draft ESSA rules allow an N-size above 30 to be chosen, but the state must justify the decision. The proposed rules clarify that the determination must be statistically sound, the same for all subgroups and sufficient to not reveal any personally identifiable information. States must describe the N-size on the report cards, and the state plan must demonstrate how it meets the regulatory requirements.
HOW IS N-SIZE CURRENTLY ADDRESSED? Ohio currently uses 30 tested students as the minimum number required to form a rated subgroup. Students who are potential test takers, but do not take the test, are not included in this minimum count. More information about Ohio s current implementation of Annual Measurable Objectives (AMOs) is available here. The Ohio Department of Education generally uses 10 as the minimum threshold for aggregate publicly reported student data. This maximizes the policy of transparency of the information while maintaining the confidentiality of students. WHAT DOES THE DATA SAY ABOUT N-SIZE IN OHIO? State Level Ohio is among 23 states that have a minimum N-size of 30 or greater. Some of those states have provisions that reduce the N-size for small schools. The U.S. Department of Education has indicated that increasing to more than 30 students would require specific information explaining why this is necessary. There is strong indication that a request to increase the N-size above 30 students would not be accepted. The following table looks at what percentage of students, in each subgroup statewide, would be included in the accountability system based on N-size determinations. Decreasing the N-size would include more students in their respective subgroups statewide. This is especially pronounced with students with disabilities and English learners (ELs), as well as Black, Hispanic, multiracial and Asian- Pacific Islander students. For example, only 51.8 percent of ELs and 51.5 percent of Hispanic students statewide are included in their school subgroup analysis with the current policy of N-size equaling 30. Adjusting the N-size to 10 would increase those numbers to 80.3 percent and 82.6 percent respectively. Subgroup All Students Students with disabilities Econ. Disadvant aged English learners White Black Hispanic Multiracial Asian-PI American Indian Total Tested 875503 128821 422402 21495 645361 130733 40161 39914 18265 1069 N- Size 10 100. 98.8% 99.9% 80.3% 99.8% 96.8% 82.6% 81.6% 73.7% 2.8% 15 99.9% 96.1% 99.7% 71.9% 99.7% 95. 72.6% 68.6% 62.8% 1.5% 20 99.9% 91.9% 99.3% 64.1% 99.6% 93.1% 64.3% 56.4% 54.7% 0. 25 99.9% 85.6% 98.9% 58.4% 99.5% 91.3% 57.1% 45. 48.7% 0. 30 99.8% 78.3% 98.2% 51.8% 99.2% 89.7% 51.5% 37.6% 43.6% 0. Using a benchmark of 95 percent of students statewide included in their schools subgroup analysis, we can demonstrate how different N-sizes have different impacts. The Green shows if/where the 95 percent threshold is met (or the highest simulated base for this analysis). Red cells are percentages based on current policy that do not meet that threshold. PAGE 2 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
The data can be plotted to see how each subgroup is included at each possible N-size. The following chart looks at the percent of economically disadvantaged students, students with disabilities and English learners. Percent of Group Evaluated 10 9 8 7 6 5 4 0 10 20 30 40 50 Minimum N-Size Economic Disadvantage Students w/ Disabilities English Language Learners Similarly, this chart shows the same trends for Black, Hispanic, Asian-Pacific Islander and multiracial students at the school level. Percent of Group Evaluated 10 9 8 7 6 5 4 0 10 20 30 40 50 Minimum N-Size Black Hispanic Asian-PI Multiracial The trend is consistent. The lower the N-size, the more students get included at the school level. PAGE 3 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
District Level The district level data (all public districts) shows that decreasing the N-size would have a corresponding increase to the number of subgroups evaluated in districts. This data is displayed in the following chart. 5 45% 4 35% 25% 15% 5% All Districts The Ohio Department of Education utilizes a district typology to analyze demographically similar districts. Ohio s large, urban districts (commonly referred to as the Ohio 8) are currently being evaluated on most subgroups. The following table displays that information. 7 Ohio 8 Districts 6 5 4 PAGE 4 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
Decreasing the minimum N-size would have an impact in other areas of the state, including small rural schools. For example, Type 3 districts (which tend to be small towns in rural counties, with low levels of racial/ethnic diversity and poverty) would have more student groups evaluated as the N-size decreases. 7 Type 3 Districts 6 5 4 This can also be shown with graduation data at the district level. 5 45% 4 35% 25% 15% 5% All Districts PAGE 5 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
School Level The school level analysis shows similar trends. As displayed below, decreasing the N-size would have a corresponding increase to the number of subgroups evaluated in schools. 45% 4 35% 25% 15% 5% All Schools Type 6 suburban districts are impacted the most by potential changes in N-size. Decreasing the N- size would lead to many more subgroups being evaluated in suburban schools. Schools in Type 6 Districts 25% 15% 5% PAGE 6 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
Due to their small student populations, many community schools are not evaluated at all. Decreasing the N-size would increase the number of schools evaluated, and the number of subgroups evaluated. 35% Community Schools 25% 15% 5% PAGE 7 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
WHAT ARE THE OPTIONS AND THE IMPLICATIONS? Ohio must include in its state plan a determination of N-size. This number must ensure equity of all students while protecting student privacy. Based on the data that has been reviewed, three options are discussed. 1) Status Quo: N=30 This is the current N-size and would not result in a change to the accountability system. Given the ESSA emphasis on subgroup inclusion, this option likely does not meet spirit of the law. Under the current determination, a significant number of schools are only evaluated (mathematics and English language arts) for the All Students and White, non-hispanic subgroups. Over 20 percent of community schools have fewer than 30 students in tested grades and therefore are not rated at all for AMO. Among dropout prevention and recovery schools, nearly 40 percent have fewer than 30 students in tested grades. 2) N=20 This option significantly increases the inclusion of the students with disabilities subgroup, as well as English learners, Hispanic, Asian and multiracial subgroups. Some subgroups remain below 70 percent participation. o o o The most significant impact of reducing N-size from 30 to 20 on assessments is with the following groups: Multiracial. 38 percent inclusion (N=30) to 56 percent inclusion (N=20): 18 percent increase Hispanic. 51 percent to 64 percent: 13 percent increase Students with disabilities. 80 percent to 92 percent: 12 percent increase English learners. 52 percent to 64 percent: 12 percent increase Asian-Pacific Islanders. 44 percent to 55 percent: 11 percent increase The impact in graduation analysis is with the following groups: Students with disabilities. 56 percent to 74 percent: 18 percent increase English learners: 25 percent to 41 percent: 16 percent increase Multiracial. 20 percent to 36 percent: 16 percent increase Hispanic. 33 percent to 48 percent: 15 percent increase Asian-Pacific Islanders. 27 percent to 40 percent: 13 percent increase More subgroups would be evaluated in more districts and schools Added Subgroups Number of Districts Number of Schools 0 422 1813 1 141 1196 2 40 260 3 5 70 4 1 4 5 1 PAGE 8 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
3) N=10 This option significantly increases all subgroups with the exception of American Indian. (Ohio population of American Indian students is too small to create subgroups except in two schools). This increases the modal number of school subgroups evaluated from three to four. o The most significant impact of reducing from 30 to 10 on assessments is with the following groups: Multiracial. 38 percent to 82 percent: 44 percent increase Hispanic. 51 percent to 83 percent: 32 percent increase Asian-Pacific Islanders. 44 percent to 74 percent: 30 percent increase English learners. 52 percent to 80 percent: 28 percent increase Students with disabilities. 80 percent to 99 percent: 19 percent increase o o The impact in graduation analysis is with the following groups: Multiracial. 20 percent to 61 percent: 41 percent increase Asian-Pacific Islanders. 27 percent to 64 percent: 37 percent increase English learners. 25 percent to 61 percent: 36 percent increase Students with disabilities. 56 percent to 92 percent: 36 percent increase Hispanic. 33 percent to 67 percent: 34 percent increase More subgroups would be evaluated in more districts and schools Added Subgroups Number of Districts Number of Schools 0 202 643 1 209 1266 2 142 813 3 46 430 4 10 153 5 31 6 7 7 1 To get a sense of the practical impact of these decisions, the following table represent data from a K- 4 school that gained seven subgroups based on this analysis. Group FY15 Enrollment All students 185 SWD 25 Econ. Disadvantaged 25 LEP 15 White 123 Black 12 Hispanic 11 Multiracial 11 Asian-Pacific Islander 28 American Indian 0 Typology 6 PAGE 9 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016
WHAT ARE THE RELATED ESSA ISSUES? Ohio also needs to review and, possibly, revise its Gap Closing measure, as well as develop a measure of English language proficiency. Both of these measures will be impacted by the N- size determination. It is important that the Gap Closing measure fairly and meaningfully distinguish school performance and give credit for improvement. Statistical validity As the group size approaches 10, the variability caused by each student result increases. For a group size of 20, each student contributes 5 percent to the overall result. For a group size of 10, that impact doubles to 10 percent. Variability of group sizes within a school With more groups being evaluated, there will be more variability among the groups in the range of sizes. This has an impact on the relative contribution of each group to the overall Gap Closing rating for the school. Related uses of minimum N-size within ESSA o The minimum participation rate allowed (without demotion) is 95 percent. Currently, the threshold for evaluating participation that the department uses is 40, which allows the possibility that at least two students in a school/subgroup can be non-test takers before the participation penalty is triggered. WHAT WAS THE METHODOLOGY FOR THIS ANALYSIS? The Ohio Department of Education s analysis of the potential impact to N-size change used a simplified model of which students factored into the AMO calculation, i.e., students in grades 3-8 and 10 for whom a school or district were accountable in academic year 2015. Notably, this initial analysis does not incorporate all students used in the actual AMO calculation, such as those who took applicable end-of-course high school assessments or those in the cohort graduation rate for 2014. Also, this analysis does not exclude students who, for any reason, were untested or had invalid scores. PAGE 10 ESSA: MINIMUM N-SIZE FOR SUBGROUP EVALUATION July 27, 2016