Second Generation Accountability for ELLs: Connecting Title I and Title III Assessment and Accountability Systems

Second Generation Accountability for ELLs: Connecting Title I and Title III Assessment and Accountability Systems David J. Francis University of Houston Texas Institute for Measurement, Evaluation and Statistics

Overview Review background on English Language Learners What factors make ELLs unique as a subgroup under NCLB? Examine the link between language proficiency and content area mastery What are the relative contributions of time in the U.S. and language proficiency to content area achievement? Propose Some Modifications to Current ELL Accountability Do the links between Time, Language Proficiency and Achievement suggest specific improvements to the way that we measure and report on the academic progress of ELLs?

Who Are English Language Learners? Over 9M LM students, roughly 5.5M classified as LEP Comprise one of the fastest-growing groups among the schoolaged population in this nation 169% from 1979 to 2003 (vs. 12% growth in general) Expected to be 30% of school-aged population in 2015 19 states have reported 10-year growth in excess of 200% Largest and fastest growing segment of ELL population is Students who immigrated before Kindergarten, and U.S. born children of immigrants

Definitions: At school entry Identification Home survey Language proficiency tests Other input (e.g., teachers) Monitoring ELLs (or LEP) Language Prof. Tests IFEP Language Title III Achievement Title I

Academic Performance Indicators for ELLs Compared to native English-speaking peers on Grade 4 NAEP, ELLs were 1/4 th as likely to score proficient or above in Reading 1/3 rd as likely in Math ELLs also perform more poorly on State tests For example, in 2002, only 18.7% of designated ELLs scored proficient in reading on state tests (9 states did not report)

ELLs as a Subgroup Under NCLB Such comparisons from State and Federal accountability systems may bias results against ELLs Unlike all other demographic groupings (e.g., gender, ethnicity, learning disabilities), membership in the ELL category is dynamic Moreover, the defining characteristic (i.e., language proficiency) is causally linked to the outcomes of interest (i.e., content area achievement) As students become proficient in English, they no longer count as members of the group

What is unique about ELLs under NCLB? Unlike other subgroups (e.g., FRL, gender, ethnicity), there is no universal definition of LEP, nor is there a universally accepted approach to identifying children as LEP. Confusion over grammaticality and complexity of thought / capacity for analytical thought Failure to distinguish conversational and academic language Failure to distinguish oral from written language Defining the population of interest and monitoring their academic progress is less precise than it needs to be States differ in the instruments used to assess language proficiency States differ in the criteria used to judge proficiency Too often, subjective judgments are part of the decision process

What is unique about ELLs under NCLB? NCLB presumes static group membership. Membership in the LEP subgroup is dynamic, if schools are successful Students are placed in the group when language proficiency is low Students lose their membership as they acquire English, Acquisition of English is a consequence of effective schooling, and mediates the effects of schooling on content mastery Regardless of the specific definition used, membership requires LOW performance on a dimension of skill (viz. Language) that is causally linked to the outcomes of interest

Definitions: Changes in Group Membership Over Time RFEP Language Prof. Tests ELLs (or LEP) IFEP

RFEP RFEP (excluded in comparisons) ELL ELL (included in comparisons)

Comparison of ELLs and former ELLs on State Reading Test in Texas 2002 Level of Language Proficiency for ELL Groups Grade Beginning Intermediate Advanced (2002) Advanced (2000) 3 13.9 38.3 90.6 90.0 4 13.1 37.4 84.1 93.6 5 16.5 24.1 69.5 96.1 6 14.5 12.8 46.0 86.8 7 15.0 12.4 43.9 85.0 8 23.2 19.2 55.3 90.2 10 21.3 28.5 66.4 85.8 Overall 15.8 30.4 76.4 89.6 http://www.tea.state.tx.us/student.assessment/reporting/results/rpteanalysis/2002/reading/statewide.html

Comparison of Graduation Rates among ELL, Former ELLs, and Never ELLs in New York City 1 Group After four Years of High School After 7 Years of High School Current ELLs 32.6 49.5 Former ELLs 60.1 76.5 Never ELLs 54.5 70.5 1 Cohort entering grade 9 in 1996. http://www.regents.nysed.gov/2005meetings/march2005/0305emscvesidd4.html

Current Law Allows retention of ELL label for accountability purposes for up to two years after achieving FEP status Makes us happy because it boosts the performance of the ELL group Gives a more accurate view of the performance of ELLs than when FEPs are not counted as ELLs

Including FEPs in the ELL Group: Are we really getting at the right information? Grade Overall %Proficient without FEP-2 1 Overall %Proficient with FEP-2 1 3 25.4 68.6 4 31.9 65.6 5 33.5 61.4 6 21.2 47.3 7 22.1 45.1 8 30.7 55.1 10 39.2 62.4 Which column best captures the long term results for ELLs? Which one really tells us how the school/district/state is doing? 1 Hypothetical result based on 2003 percentages in each language proficiency category http://www.tea.state.tx.us/student.assessment/reporting/results/rpteanalysis/2002/reading/statewide.html

Retaining FEPs in the ELL group Improves typical practice, but is insufficient to get an accurate and complete picture about the performance of ELL students It does not allow reporting of long term outcomes for students who began school as ELL, and Confounds language proficiency and achievement

Including FEPs in the ELL Group: Are we really getting at the right information? Level of Language Proficiency for ELL Groups Grade Beginning Intermediate Advanced (2002) Advanced (2000) Overall %Proficient without FEP-2 1 Overall %Proficient with FEP-2 1 3 13.9 38.3 90.6 90.0 25.4 68.6 4 13.1 37.4 84.1 93.6 31.9 65.6 5 16.5 24.1 69.5 96.1 33.5 61.4 6 14.5 12.8 46.0 86.8 21.2 47.3 7 15.0 12.4 43.9 85.0 22.1 45.1 8 23.2 19.2 55.3 90.2 30.7 55.1 10 21.3 28.5 66.4 85.8 39.2 62.4 Which column best captures the long term results for ELLs? Which one really tells us how the school/district/state is doing? 1 Hypothetical result based on 2003 percentages in each language proficiency category http://www.tea.state.tx.us/student.assessment/reporting/results/rpteanalysis/2002/reading/statewide.html

Retaining FEPs in the ELL group Improves typical practice, but is insufficient to get an accurate and complete picture about the performance of ELL students It does not allow reporting of long term outcomes for students who began school as ELL, and Confounds language proficiency and achievement

What would you conclude if the overall results looked like this from 2002 to 2003? Level of Language Proficiency for ELL Groups Grade Beginning Intermediate Advanced (2002) Advanced (2000) Overall %Proficient 2002 1 Overall %Proficient 2003 3 68.6 68.6 4 65.6 64.9 5 61.4 60.7 6 47.3 44.6 7 45.1 42.3 8 55.1 52.2 10 62.4 62.7 1 Hypothetical result based on 2003 percentages in each language proficiency category http://www.tea.state.tx.us/student.assessment/reporting/results/rpteanalysis/2002/reading/statewide.html

Aggregate Reporting Masks Performance Changes when Demographics Shift Level of Language Proficiency for ELL Groups Grade Beginning Intermediate Advanced Advanced 2 Years Prior Overall %Proficient 1 3 15.3 42.1 92.4 91.8 68.6 4 14.4 41.1 85.8 95.5 64.9 5 18.2 26.5 70.9 98.0 60.7 6 15.9 14.1 46.9 88.5 44.6 7 16.5 13.6 44.8 86.7 42.3 8 25.5 21.1 56.4 92.0 52.2 10 23.4 31.4 67.7 87.5 62.7 The percentage of children in each language proficiency category is partly a function of instruction, but it also a function of demographics. Performance is up for children in each category of language proficiency, but overall performance is down because of changes in the percentage of students in each proficiency category. 1 Hypothetical result based on increasing achievement in all groups while increasing the percentage of students in the lowest three categories of language proficiency

Allowing FEP students to count in the ELL category for up to two years Boosts the overall percent proficient within the ELL category, But it does NOT allow us to determine the academic achievement of ELLs who become proficient in English allow us to determine the long term achievement outcomes for children who entered school as ELLs provide schools with actionable information about their ELL students performance, or solve the problem of aggregation bias when demographics are shifting

Current accountability practice for ELLs Provides an overly simplistic view of a complex developmental and educational process Fails to take into account the developmental nature of language acquisition Language takes time to acquire This time frame contains both maturational and environmental influences It can be accelerated through good instruction and experiences in a language rich environment, but it cannot be reduced to zero Fails to take into account the causal role that language plays in acquisition of content area knowledge

How do time and language work to predict the content area achievement of ELLs?

Assessment of Language Proficiency under NCLB (Title III) Must include Listening, Speaking, Reading, and Writing Used for Title III accountability, but Also commonly used for placement, re-designation, decisions about interventions, etc. Must be uniform within state for all students covered by Title III, but is not required for ELL students not receiving Title III services Language proficiency assessments vary greatly across states in content and format

High Stakes Assessments and ELLs The effect of language proficiency on ELL performance on content area assessments is a matter of considerable interest to educators, as well as to students and their families. At the same time, this question has not been widely studied in the contexts of today s Title I and Title III assessments Title I assessments - content area assessments Title III assessments - measure English language acquisition Reading, Writing, Speaking, and Listening

How are Years in the US, and ELP performance related to performance on the ELA and Math content assessment? ELP Assessment Subtests of Reading (0-30), Writing (0-30), Speaking (0-30), Listening (0-5) Composite Scaled Score (300-400) Performance Levels 1 Beginning (300-324) Early Intermediate (325-348) Late Intermediate (349-374) Transitioning (375-400) 1 Proficiency cut points vary slightly by grade

Sample Sizes for state-level data base Cluster Type N Number with 10 or Fewer Students Maximum N Schools 948 573 304 Districts 170 67 2,590

Student Demographics Grade Demographic ALL 4 5 6 7 8 %Female 47% 48% 47% 46% 48% 55% %Asian a 18% 19% 17% 17% 17% 17% %Black a 11% 10% 11% 10% 11% 12% %Hispanic a 58% 57% 58% 59% 60% 60% %Native American a <1% <1% <1% <1% <1% <1% %Caucasian a 13% 14% 14% 14% 12% 12% %Title I 71% 78% 75% 71% 62% 63% %Free/Reduced Lunch 81% 81% 80% 81% 81% 82% %Special Education 20% 19% 22% 21% 20% 17% %Proficient ELP c 46% 60% 40% 44% 36% 44% Sample Size b 17,767 4,663 3,566 3,303 3,172 3,063

14.2% 13.6% 13.0% 15.8% 17.1% 17.1% 10.7% 18.2% 16.5% 22.6% 11.6% 10.5% 11.6% 12.4% 18.9% 12.2% 5.7% 8.8% 7.6% 13.0% 6.9% 5.5% 6.8% 7.4% 12.7% Proficient

15.1% 12.6% 7.9% 7.1% 5.5% 19.3% 13.6% 13.7% 10.8% 11.6% 14.6% 15.2% 12.5% 10.0% 13.3% 15.3% 14.4% 13.2% 8.8% 10.6% 10.0% 12.7% 10.8% 9.9% 12.5% Proficient

ELA and Math Performance by ELP Category Grade ELP Performance ELA MATH N MEAN SD %Pass %Prof N MEAN SD %Pass %Profs 4 1 209 209.90 6.47 3.8% 0.5% 392 211.38 7.81 11.5% 1.0% 2 287 213.80 5.30 11.2% 0.6% 343 215.47 9.34 23.0% 2.6% 3 977 218.79 6.70 32.6% 1.9% 1019 220.15 10.15 42.8% 5.2% 4 2674 230.26 11.59 80.5% 20.9% 2696 230.68 14.08 77.3% 22.3% 5 1 295 211.93 6.05 7.5% 0.6% 449 210.46 6.59 6.5% 0.8% 2 370 214.98 6.26 20.3% 0.7% 420 214.89 9.55 21.0% 3.2% 3 1093 220.73 8.09 46.0% 2.3% 1138 219.12 11.40 32.0% 7.9% 4 1332 232.09 11.08 86.6% 25.3% 1352 229.69 15.93 64.3% 25.8% 6 1 289 211.70 6.53 8.7% 0.2% 422 210.48 6.90 5.9% 1.4% 2 311 214.60 5.48 13.8% 0.9% 340 213.82 8.60 14.1% 2.3% 3 913 219.87 7.51 43.4% 2.6% 964 217.34 10.30 22.6% 5.2% 4 1358 231.19 10.75 86.3% 25.3% 1378 226.75 14.96 55.0% 19.4% 7 1 413 210.03 5.74 2.7% 0.8% 563 210.50 7.20 6.4% 1.5% 2 362 214.17 5.21 10.8% 0.8% 387 213.56 8.57 10.6% 3.3% 3 887 219.60 6.95 39.2% 2.3% 906 216.72 9.76 21.6% 4.4% 4 1067 232.90 10.82 89.9% 33.1% 1080 226.68 14.72 56.6% 19.4% 8 1 414 212.57 5.16 5.6% 0.2% 503 210.33 5.48 4.6% 0.4% 2 296 214.99 5.49 13.5% 0.6% 325 213.11 8.93 10.2% 3.3% 3 748 220.17 7.71 42.1% 4.0% 767 215.42 9.07 16.3% 3.7% 4 1259 233.24 11.52 87.7% 35.6% 1274 224.76 14.56 48.8% 18.9% a %Pass is the percentage of students scoring above the state s predefined minimum passing standard of 220 for the ELA and MATH assessments. Proficiency is defined as scoring above 240.

Proficient Proficient

Proficient Proficient

3-Level Model for ELA and Math Unconditional Model (within grade) V(Students(schools)) V(Schools(Districts)) V(Districts) Conditional Models Years in US ELP Years in US and ELP

3-Level Model for ELA and Math Conditional Models Model A Years in US ELP measured as Performance Levels Years in US + ELP measured as Performance Levels Model B Years in US + ELP measured as Performance Levels ELP measured as a composite Scaled Score ELP measured as Domain Scores (R, W, S, L) ELP measured as Domain Scores (Reading, Writing)

Unconditional Random Effects for ELA and MATH ELA MATH Grade Source Estimate s.e. Z p %Variance a Estimate s.e. Z p %Variance a 4 District 23.99 5.98 4.01 <.0001 0.15 36.70 8.18 4.49 <.0001 0.17 Schools 25.61 3.30 7.76 <.0001 0.16 34.10 4.26 8.00 <.0001 0.15 Students 111.69 2.62 42.70 <.0001 0.69 149.64 3.37 44.36 <.0001 0.68 5 District 25.20 5.95 4.23 <.0001 0.17 43.69 9.35 4.67 <.0001 0.19 Schools 15.58 2.92 5.33 <.0001 0.11 34.94 5.32 6.57 <.0001 0.15 Students 107.48 2.94 36.57 <.0001 0.72 151.28 3.98 38.01 <.0001 0.66 6 District 21.05 6.20 3.39 0.0003 0.15 48.79 11.27 4.33 <.0001 0.23 Schools 20.47 4.04 5.06 <.0001 0.14 23.93 5.30 4.51 <.0001 0.11 Students 100.77 2.82 35.77 <.0001 0.71 135.55 3.67 36.96 <.0001 0.65 7 District 25.79 6.72 3.84 <.0001 0.17 58.80 12.91 4.55 <.0001 0.29 Schools 17.57 3.73 4.72 <.0001 0.12 20.00 4.37 4.58 <.0001 0.10 Students 108.15 3.06 35.36 <.0001 0.71 120.66 3.30 36.56 <.0001 0.60 8 District 26.05 7.63 3.41 0.0003 0.16 52.35 11.19 4.68 <.0001 0.27 Schools 24.18 5.11 4.73 <.0001 0.15 29.67 5.43 5.47 <.0001 0.15 Students 115.44 3.28 35.23 <.0001 0.70 110.01 3.03 36.27 <.0001 0.57 a %Variance computed as intra-class correlations (ICCs), viz. as ratio of estimate to sum of estimates for District, School, and Students. Percentages may not sum to 100% due to rounding.

Conditional Random Effects for ELA and MATH predicted from Years in US, ELP, and Years + ELP Grade 4 5 6 7 8 Source ELA MATH Years in ELP- Years Years in ELP- Years and US ΔR 2 Perf. ΔR 2 and ELP ΔR 2 US ΔR 2 Perf. ΔR 2 ELP ΔR 2 District 27.21-0.13 15.13 0.37 14.73 0.39 41.11-0.12 29.10 0.21 26.51 0.28 Schools 25.04 0.02 15.66 0.39 15.72 0.39 32.74 0.04 22.62 0.34 22.85 0.33 Students 108.37 0.03 81.83 0.27 81.67 0.27 145.14 0.03 119.72 0.20 118.84 0.21 District 25.73-0.02 11.62 0.54 11.11 0.56 45.24-0.04 36.52 0.16 35.45 0.19 Schools 14.83 0.05 9.25 0.41 9.53 0.39 33.28 0.05 23.34 0.33 22.88 0.35 Students 104.37 0.03 70.30 0.35 69.65 0.35 149.57 0.01 120.02 0.21 117.86 0.22 District 22.15-0.05 9.16 0.56 8.59 0.59 49.56-0.02 35.88 0.26 31.27 0.36 Schools 18.24 0.11 12.68 0.38 12.90 0.37 23.81 0.01 20.05 0.16 20.34 0.15 Students 97.03 0.04 66.38 0.34 66.07 0.34 133.72 0.01 111.82 0.18 109.45 0.19 District 27.88-0.08 11.20 0.57 11.05 0.57 61.72-0.05 47.32 0.20 43.68 0.26 Schools 13.08 0.26 4.53 0.74 4.63 0.74 19.42 0.03 14.44 0.28 15.03 0.25 Students 104.51 0.03 60.65 0.44 60.68 0.44 119.63 0.01 97.70 0.19 95.91 0.21 District 26.70-0.02 10.87 0.58 9.54 0.63 51.31 0.02 42.76 0.18 37.47 0.28 Schools 22.99 0.05 7.58 0.69 8.46 0.65 30.17-0.02 22.09 0.26 21.84 0.26 Students 113.83 0.01 73.83 0.36 72.40 0.37 109.00 0.01 92.13 0.16 89.03 0.19 a ΔR 2 computed as change in variance component from unconditional model (Table 5) relative to magnitude of variance component in unconditional model (Table 5-Table 6)/(Table 5).

Conditional Random Effects for ELA and MATH predicted from Years in US + ELP measured as (1) Performance Levels, (2) Scaled Score, or (3) Domain Scores Grade Source ELA MATH Years + Years + Years + Years Years + Years and ELP-PL ΔR 2 ELP-SS ΔR 2 ELP-DS ΔR 2 ELP-PL ΔR 2 ELP-SS ΔR 2 ELP-DS ΔR 2 4 District 14.73 0.39 14.00 0.42 10.92 0.54 26.51 0.28 25.27 0.31 20.12 0.45 Schools 15.72 0.39 14.90 0.42 12.81 0.50 22.85 0.33 22.08 0.35 18.65 0.45 Students 81.67 0.27 74.28 0.33 60.20 0.46 118.84 0.21 112.80 0.25 100.78 0.33 5 District 11.11 0.56 10.72 0.57 7.66 0.70 35.45 0.19 33.94 0.22 25.35 0.42 Schools 9.53 0.39 8.27 0.47 7.21 0.54 22.88 0.35 22.44 0.36 20.03 0.43 Students 69.65 0.35 65.37 0.39 60.02 0.44 117.86 0.22 112.85 0.25 105.12 0.31 6 District 8.59 0.59 7.02 0.67 7.60 0.64 31.27 0.36 28.26 0.42 26.63 0.45 Schools 12.90 0.37 10.78 0.47 6.67 0.67 20.34 0.15 18.99 0.21 17.36 0.27 Students 66.07 0.34 61.97 0.39 56.49 0.44 109.45 0.19 104.52 0.23 97.72 0.28 7 District 11.05 0.57 11.09 0.57 8.04 0.69 43.68 0.26 42.19 0.28 34.16 0.42 Schools 4.63 0.74 4.05 0.77 3.06 0.83 15.03 0.25 14.21 0.29 12.25 0.39 Students 60.68 0.44 57.85 0.47 53.16 0.51 95.91 0.21 93.23 0.23 85.99 0.29 8 District 9.54 0.63 8.36 0.68 3.32 0.87 37.47 0.28 35.03 0.33 27.24 0.48 Schools 8.46 0.65 7.14 0.70 5.52 0.77 21.84 0.26 20.94 0.29 19.99 0.33 Students 72.40 0.37 69.24 0.40 60.64 0.47 89.03 0.19 85.41 0.22 75.99 0.31 a ΔR 2 computed as change in variance component from unconditional model (Table 5) relative to magnitude of variance component in unconditional model (Table 5-Table 7)/(Table 5).

Comparison of Random Effects Using Different Approaches to Measuring ELP Random Effects (Years in US Plus ELP Measured in Different Ways) Grad ELA MATH e Source ELP SS DS RW ELP SS DS RW 4 District 14.7 14.0 10.9 10.6 26.5 25.3 20.1 21.1 Schools 15.7 14.9 12.8 13.5 22.9 22.1 18.7 18.9 Students 81.7 74.3 60.2 63.8 118.8 112.8 100.8 103.5 5 District 11.1 10.7 7.7 10.3 35.5 33.9 25.4 30.7 Schools 9.5 8.3 7.2 6.1 22.9 22.4 20.0 21.1 Students 69.7 65.4 60.0 64.7 117.9 112.9 105.1 109.1 6 District 8.6 7.0 7.6 8.4 31.3 28.3 26.6 30.3 Schools 12.9 10.8 6.7 6.0 20.3 19.0 17.4 17.5 Students 66.1 62.0 56.5 60.4 109.5 104.5 97.7 101.5 7 District 11.1 11.1 8.0 7.8 43.7 42.2 34.2 34.5 Schools 4.6 4.1 3.1 4.1 15.0 14.2 12.3 14.9 Students 60.7 57.9 53.2 56.8 95.9 93.2 86.0 89.0 8 District 9.5 8.4 3.3 3.3 37.5 35.0 27.2 32.5 Schools 8.5 7.1 5.5 6.3 21.8 20.9 20.0 20.1 Students 72.4 69.2 60.6 65.9 89.0 85.4 76.0 80.7

Analysis Summary Years in the US predicted ELA and MATH performance at the district, school, and student levels However, Years in the US was a relatively weak predictor compared with ELP When ELP was included with Years in US, the effects of Years in the US were unsystematic and small. Effects of ELP remained strong and consistent (i.e., outcomes increased with increases in ELP)

Analysis Summary How ELP was measured made some difference in its value as a predictor; Domain Scores predicted best Using Domain Scores for Reading and Writing only was almost as good as using Reading, Writing, Speaking, and Listening These results suggest that the academic components of the language assessment are the most important predictors of content area achievement It is noteworthy that ELP performance accounted for so much of the school and district variability in ELA and MATH

Conclusions from Statistical Analyses Taken together, the results highlight the importance of language in the development of content area knowledge. They further highlight that children need to be taught the content in order to close achievement gaps. Gaining proficiency in English was not a guarantee of success on the content area assessments.

Suggestions for Improved Accountability for ELLs We clearly have a reporting problem that is fueled by the dynamic nature of the ELL category and the role that language plays in the development of content area proficiency Create a separate reporting category of ELLs who are reclassified as FEP Report achievement results within ELP proficiency levels Beginner, Intermediate, Advanced Intermediate, Fluent English Proficient

Suggestions for Improved Accountability for ELLs (cont.) Set achievement performance expectations that are challenging but realistic for each particular group Expect students in the FEP category to perform at levels comparable to monolinguals in the school/district/state Establish more realistic achievement targets for students who are still developing English Goals must be challenging, but attainable, to properly motivate students and teachers Long term expectations for all students are the same, but short term goals must be challenging, yet attainable in the short term

Suggestions for Improved Accountability for ELLs (cont.) Hold schools/districts accountable under Title I for language proficiency Hold schools/districts accountable under Title I for the time required for children to achieve FEP status Expect all students to reach FEP status Short-circuit gamesmanship by limiting the time before children must be counted in the FEP category for accountability

How do these steps improve accountability? Gives schools actionable information about the content area achievement of ELLs Allows more accurate evaluation of school performance in the face of shifting demographics Credits language proficiency gains and content area achievement gains that are reasonable given students language proficiency Meaningfully counts the achievement results for all children regardless of their level of language proficiency Recognizes the importance of setting challenging but attainable goals to motivate maximal performance

Conclusions An accountability model that addresses these issues will provide more accurate information to teachers, principals, and other stakeholders about the performance of ELLs Place emphasis on integration of language instruction into content area instruction, and Increase the emphasis on teaching content when ELLs first reach school Increase the demand for language tests that will serve as better barometers of ELL students acquisition of the academic language skills needed to master content domains.

Thank You dfrancis@uh.edu