Visualizing Educational Data

Visualizing Educational Data Michael Marder Department of Physics, The University of Texas at Austin February 9, 211 1 Preliminaries Term Well-Off Low-Income Poverty Concentration Free and Reduced Lunch Definition Student has never been eligible for free and reduced lunch Student who is or has been eligible for free and reduced lunch Percentage of students in school currently eligible for free and reduced lunch 1. Students are eligible for reduced price meals when their family income is 185% of the poverty guideline, and for free meals when their family income is 13% of the poverty guideline. The poverty guidelines for 21 appear below Number in Family Poverty Guideline Number in Family Poverty Guideline 1 1,83 5 25,79 2 14,57 6 29,53 3 18,31 7 33,27 4 22,5 8 37,1 NAEP National Assessment of Educational Progress, The Nation s Report Card. Assessments of reading and mathematics administered to a sample of around 15,-2, US students in 4th and 8th grade roughly every two years. More infrequently there are exams at 12th grade, and exams in science and social studies. TIMSS Trends in International Mathematics and Science Study. PISA TAKS Programme for International Student Assessment Texas Assessment of Knowledge and Skills. The Texas high-stakes exam that provides data both for the Texas school accountablity system and for the federal accountability system under No Child Left Behind. Mathematics and Reading are tested every year for students from third to eleventh grade, with exceptions for students with language and learning disabilities. Texas SAT A score of 111 on the SAT or 24 on the ACT. Texas uses for SAT only mathematics and critical reading 2. Criterion Commended A score on the mathematics TAKS exam that varies by grade level and year but is approximately 9% Table 1: Definitions of terms 1

2 2 INTERNATIONAL COMPARISONS 2 International Comparisons PISA--Programme for International Student Assessment Among 3 developed countries, the United States is ranked twenty-fifth in math and twenty-first in science. When the comparison is restricted to the top 5 percent of students, the United States is ranked last. -- Davis Guggenheim (21), Waiting for Superman PISA Mathematics 26, 15-Year-Olds Mathematics Scaled Score Mathematics Scaled Score 55 5 45 4 35 55 5 45 4 35 Well-Off Finland Belgium Switzerland Korea Netherlands New Zealand Japan Canada Germany Czech Rpblc Australia Hungary Denmark France Austria Slovak Repblc Luxembourg Poland Sweden UK Ireland Iceland Portugal Norway United States Spain Greece Italy Turkey Mexico Argentina Low-Income Finland Korea Canada Netherlands Australia Switzerland Japan Denmark Poland Belgium New Zealand Czech Rpblc Iceland Sweden Ireland Germany Slovak Rpblc Hungary Austria Norway UK Spain France United States Luxembourg Italy Portugal Greece Turkey Mexico Argentina Figure 1: Mathematics proficiency of 15-year-olds comparing various countries. Family economic status is indicated by whether father is low-skill blue-collar worker or high-skill white collar worker. Comments: This exam focuses on students of a specific age -- 15 years old -- rather than on a specific grade level. It emphasizes practical application of knowledge over familiarity with procedures and vocabulary. The US indeed ranks very low.

3 TIMSS--Trends in International Mathematics and Science Study TIMSS--Trends in International Mathematics and Science Study TIMSS Mathematics Eighth Grade Mathematics Scaled Score 6 55 5 45 4 35 27 1995 Taiwan South Korea Singapore Hong Kong Japan Hungary England Russian Federation United States Lithuania Czech Republic Slovenia Armenia Australia Sweden Malta Scotland Serbia Italy Malaysia Norway Cyprus Bulgaria Israel Ukraine Romania Bosnia and Herzegovina Lebanon Thailand Turkey Jordan Tunisia Georgia Iran Bahrain Indonesia Syrian Arab Republic Egypt Algeria Colombia Oman Palestinian Nat l Auth. Botswana Kuwait El Salvador Saudi Arabia Ghana Qatar Figure 2: Data on eighth grade mathematics performance from TIMSS in 1995 and 27 Comments: TIMSS more closely resembles US high-stakes exams than PISA. In 27 the US tied for sixth place in eighth grade mathematics.

4 2 INTERNATIONAL COMPARISONS Physics Population Percentage 2% 15% 1% 5% % Mechanics U.S. China 5% 1% Score Population Percentage 2% Electricity & Magnetism 15% 1% 5% U.S. China % 5% 1% Score Figure 3: Comparison of US and Chinese freshmen in college physics taking conceptual physics exams. The graphs give a more complete picture than can be obtained just by reporting average scores. For example, in Electricity & Magnetism, there are almost no U.S. students who match a typical Chinese student. After Bao, L. et al Learning and Scientific Reasoning, Science, 29, 323, 586-587 Comments In much of the rest of the world it is customary for all students to have three or four years of physics in secondary school, while in the United States only 37% of high school students have as much as one year. The difference is apparent. The topics Mechanics and Electricity & Magnetism refer to the two core subjects in introductory physics. In a test of scientific reasoning where Chinese students receive no more explict instruction than US students, students in the two countries have nearly identical performance.

5 3 National Comparisons New York State s fourth and eighth graders made no notable progress on federal math exams this year, according to test scores released on Wednesday, sharply contradicting the results of state-administered tests that showed record gains...across the country, many states posted disappointing results, with fourth-grade students stagnant nationally for the first time in nearly two decades. -- Jennifer Medina, U.S. Math Tests Find Scant Gains Across New York New York Times, October 14, 29 NAEP8thGradeReadingandMathematics,2-29 State DC Hawai Nevada Alabama Misisippi WestVirginia NewMexico Tennese California Louisiana Arkansas Oklahoma RhodeIsland Arizona Michigan Florida Kentucky Georgia Utah Delaware Alaska Misouri Idaho Wyoming Maine Maryland Iowa NewHamps. Oregon Ilinois Indiana Ohio Nebraska NewYork Pennsylvania Virginia SouthCaroli. Connecticut Wisconsin Washington Colorado SouthDakota Texas NorthCaroli. NorthDakota Montana NewJersey Vermont Kansas Minnesota Masachusets Mathematics 3 28 26 24 DC Hawai Nevada California NewMexico Alabama Louisiana WestVirginia Misisippi Alaska Arizona Arkansas Florida Tennese SouthCarolina Utah Oklahoma Georgia RhodeIsland Michigan Idaho Delaware Maryland NorthCarolina Wyoming Indiana Oregon Texas Wisconsin Misouri Washington Iowa Kentucky Colorado NorthDakota Virginia NewHampshire Maine Minnesota Ilinois Nebraska Ohio SouthDakota Kansas Pennsylvania Montana Connecticut NewYork Vermont NewJersey Masachusets 28 Reading 26 24 EligibleFreeLunch NotEligibleFreeLunch Figure 4: Eighth grade scores in mathematics and reading from the National Assessment of Educational Progress (NAEP) averaged over the period 2-29. In each state the average score of students eligible for free and reduced lunch is compared with the score of those who are not.

6 3 NATIONAL COMPARISONS NAEP8thGradeMathematics,2-29 State 31 DC Hawai Nevada Misisippi Alabama NewMexico WestVirginia Arkansas California Tennesee Louisiana Oklahoma RhodeIsland Arizona Georgia Kentucky Utah Michigan Florida Misouri Wyoming Idaho Delaware Maine Alaska Maryland SouthCarolina Ilinois Oregon Virginia Indiana NewYork Ohio Iowa Nebraska NewHampshire Texas Pennsylvania NorthCarolina Connecticut NorthDakota Vermont Wisconsin Washington Montana Colorado Kansas SouthDakota NewJersey Minnesota Masachusets 3 29 Mathematics 28 27 26 25 24 Year 2 23 25 27 29 EconomicStatus Wel-Of Low-Income Figure 5: Time development since 2 of 8th grade NAEP mathematics scores, displaying separately well-off and low-income students in each state. In every jurisdiction scores have risen since 2. Comments Eighth grade mathematics scores in every state have risen rather steadily in every state since 2 for both well-off and low-income eight graders. The magnitude of the rise is comparable to the differences among states, although not yet to the difference between well-off and low-income students. Data at the twelth grade are not available. For a dynamic representation of these data, see this Motion Plot.

7 Poverty and Educational Outcomes Poverty and Educational Outcomes NAEP Mathematics,23-29,8thGradeWel-Of Students NAEP Mathematics,23-29,8thGradeLow-Income Students 3 3 29 29 AverageScore 28 27 AverageScore 28 27 26 26 25 25 24 2 3 4 5 6 7 PovertyConcentration 24 2 3 4 5 6 7 PovertyConcentration MinorityPercentage 6.24 94.2 NumbersofWel-OforLow-IncomeStudents 22, 2,, 1,, 3,, 3,5, Figure 6: Mathematics scores on 8th grade mathematics, averaged over the four administrations of the National Assessment of Educational Progress since 23. Each disk is a state. For each state students are divided between well-off and low-income Comments Poverty is associated with mathematics test results, both at the individual and at the state level. Within any given state, low-income students get scores around 25 points less than well-off students. The states with the lowest concentration of poverty have scores 2 points higher both for well-off and low-income students than states with the highest concentration of poverty.

8 3 NATIONAL COMPARISONS Union Status and Educational Outcomes I believe that what is wrong with our schools in this nation is that they have become unionized in the worst possible way...this unionization and lifetime employment of K-12 teachers is off-the-charts crazy 3 -- Steve Jobs, Feb 16, 27 NAEP Mathematics,23-29,8thGradeWel-Of Students 3 NAEP Mathematics,23-29,8thGradeLow-Income Students 3 29 29 AverageScore 28 27 AverageScore 28 27 26 26 25 25 24 2 3 4 5 6 7 PovertyConcentration 24 2 3 4 5 6 7 PovertyConcentration UnionStatus WeakerUnions StrongerUnions NumberWel-OforLow-IncomeStudents 27,5 1,, 2,, 3,, 3,5, Figure 7: States with weak unions (Right to Work states) are highlighted for both well-off and low-income students. Each disk is a state. Comments States with and without strong unions are intermingled. Well-off students in wealthy states with strong unions have the highest outcomes. For low-income students the states with highest outcomes have weak unions. Differences in state performance that might be attributed to unions are small compared with effects of poverty. 3 http://www.chron.com/disp/story.mpl/tech/news/456691.html

9 Value-Added Methods for Assessing Teachers Value-Added Methods for Assessing Teachers Although it can be difficult to know with much certainty who is likely to be an effective teacher during a job interview, we have shown that school districts can learn a lot about teachers future effectiveness simply by scrutinizing their record during their first few years on the job...because most districts have never assembled the data required to calculate the value-added by individual teachers, the payoff to beginning to do so could be enormous. Gordon, Kane, and Staiger (26) Identifying Effective Teachers Using Performance on the Job Data from 78 randomly flipped pairs of teachers in Los Angeles, (Kane and Staiger, 28) Measured Superiority ( Years of Learning ) 4 2 Predicted Superiority ( Years of Learning ) 2 4 2 4 Most Measurements Here if Teachers are Predictably Good or Bad Half the Measurements Here if Classes are Good or Bad Independent of Teacher Figure 8: Data of Kane and Staiger for 78 pairs of teachers in Los Angeles, comparing predictions for student performance on an elementary school mathematics test with actual average student performance following random assignment of teachers to classrooms. After Kane and Staiger (28), Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation, http://www.nber.org/papers/w1467. Comments Advocates of Value-Added methods for assessing teacher performance commend this as the most careful study. It differs from other studies because pairs of teachers were randomly switched between classrooms before the school year began. Kane and Staiger report scaled scores for which they say that.25 corresponds to about a year of learning; this conversion was used to obtain the scale in this figure. They used four years of teacher and student data to form predictions about relative performance of paired teachers in a fifth year. Although measured student performance is correlated with past performance, conclusions about individual teachers were inaccurate. Seventeen percent of the time the prediction is too low by half a year of learning ( ±.125) or more Thirty-five percent of the time the prediction lies within half a year of learning of the measured value. Forty-eight percent of the time the prediction is too large by half a year of learning or more.

1 3 NATIONAL COMPARISONS Previous Year s Score is... 2-3% 3-4% 4-5% 5-6% 6-7% 7-8% 8-9% 9-1% Score Change 2 1 1 Score Changes, Texas Mathematics, 23--27 Grade 9 2 4 6 8 1 Poverty Concentration Grade 1 2 4 6 8 1 Poverty Concentration Grade 11 2 4 6 8 1 Poverty Concentration Figure 9: Average year-to-year changes in Texas student mathematics scores averaged over 23 through 27. Students are grouped according the the poverty concentration in their school, and according to their score the year before. Comments For secondary school mathematics, there are factors in addition to poverty that are strongly associated with changes in student scores. The students with the highest score gains are eleventh-graders in schools with little poverty who got 2-3% of the problems right in tenth grade, and the students with the lowest score gains are ninth-graders in schools with high poverty who got 9-1% of the problems right in eighth grade. Student gains in a teacher s class can therefore strongly depend upon the grade level and prior score of the students.

11 4 Texas Data There is no doubt that poverty presents real challenges, and it is harder to be a principal or teacher in an urban district than it is in the suburbs. But even in the toughest of neighborhoods and circumstances, children excel when the right adults are doing the right things for them. --Michelle Rhee (21), Waiting for Superman SAT College-Readiness Criterion Texas Graduates Meeting SAT Criterion (28) Percentage Meeting SAT Criterion 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Number of Graduates 1 1 5 Figure 1: Percentage of high school graduates meeting Texas SAT/ACT College Readiness Criterion plotted as a function of concentration of poverty. Every disk is a high school, with the area of the disk proportional to the number of graduates. Colors indicate the percentage of minority students in school. Comments Figure 1 is a scatter plot showing Texas high schools as a function of the fraction of students eligible for free and reduced lunch. The vertical axis shows percentage of graduates meeting the SAT/ACT College Readiness criterion in 28. Above a poverty concentration of 8% there are no examples of Texas schools where more than 2% of the graduates meet the SAT Criterion.

12 4 TEXAS DATA SATHistograms,23-39 PercentageGraduates MeetingSATCriterion 6-7% Average Number peryear 1K 5K PovertyConcentration 5-6% Average Number peryear 1K 5K 4-5% 3-4% Average Number peryear Average Numbe rper Year 1K 5K 1K 5K 2-3% 1-2% -1% Average Number peryear Average Number peryear Average Number peryear 1K 5K 1K 5K 1K 5K -2% 2-4% 4-6% 6-8% 8-1% MinorityPercent. 1. Figure 11: Histogram showing numbers of students in schools where -1%, 1-2%, etc. students met the Texas SAT/ACT College Readiness Criterion, grouped by fraction of students in the school eligible for free and reduced lunch. The data come from 23-29, and show the average numbers of students per year in each category. Comment Data such as in Figure 1 are grouped together for the period 26-29, showing numbers of students in every category of poverty concentration and SAT achievement rather than individual schools. Each bar is color coded to indicate the fraction of minority students in that category. In Texas schools where 8% to 1% of the children are poor, the odds that more than 2% will be College-Ready by the SAT criterion are zero.

13 Time Dependence of SAT College-Readiness Criterion Time Dependence of SAT College-Readiness Criterion AnnualChangeinPercentageMeetingSATCriterionversusAnnualChangeinPoverty Concentration,25-29 PercentageMeetingSATCriterionin28 5-6% 4-5% 3-4% 2-3% 1-2% -1% AnnualChangeRateSATCriterion (%/year) 1 5-5 -1-1 1 AnnualChange RatePoverty Concentration (%/year) -1 1 AnnualChange RatePoverty Concentration (%/year) -1 1 AnnualChange RatePoverty Concentration (%/year) -1 1 AnnualChange RatePoverty Concentration (%/year) -1 1 AnnualChange RatePoverty Concentration (%/year) -1 1 AnnualChange RatePoverty Concentration (%/year) CampusPovertyConcentration 2.2 99.1 TotalStudentsinCampus,29 66 2, 1, 3, 4,259 Figure 12: Comments Changes in the level of school poverty over time are correlated with changes in the Texas SAT/ACT College Readiness Criterion. The effect is most dramatic for schools with low levels of poverty, as shown in Figure 12. The plot shows all schools where both the change in poverty concentration and the change in student fraction meeting the SAT criterion over the period 26-29 are described by a linear relation with correlation coefficient r of.5 or greater. The lower the concentration of poverty, the more rapid is the change of student outcomes with changes in poverty level. This finding is consistent with the fact that the slope of the curve in Figure 1 is greatest on the left hand side of the plot.

14 4 TEXAS DATA Change of Individual Students over Time Well Off Low Income Well Off Low Income 23-24 24-25 25-26 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 Grade Grade Grade 26-27 27-28 28-29 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% 3-4 4-5 5-6 6-7 7-8 8-9 9-1 1-11 Grade Grade Grade Figure 13: Math flow plots. The vertical axis separates students according to whether or not they have ever been eligible for free and reduced lunch. The horizontal axis shows grade levels. Comment Figure 13 provides a graphical representation of the year to year changes of student mathematics scores in Texas, separating out well-off and low-income students. Each arrow shows the average score one year later of students who shared a certain starting score in the previous year. The plots provide snaphots from single year and do not show the movement of a single cohort of students. The area of an arrow is proportional to the number of students making the transition, with large arrows corresponding to tens of thousands of students. [Methods described by Marder and Bansal, PNAS, 29]

15 Change of Individual Students over Time MathemiatcsRawScore 1% 9% 8% 7% 6% 5% 4% Flowstreamlines,26-27mathematics 3 4 5 6 7 8 9 1 11 Grade FreeLunchEligibility Low-Income Wel-Of (A) year 26 27 28 Rich-PoorMathematicsGap,26-29 AnnualChangein Rich-PoorGap AnnualChangein Rich-PoorGap AnnualChangein Rich-PoorGap 2.% 1.%.% -1.% 2.% 1.%.% -1.% 2.% 1.%.% -1.% 4 5 6 7 8 9 111 Grade Figure 14: (A) Flow streamlines showing the different predicted paths of well-off and low-income students who start with the same score in third grade. (B)The rate at which the performance gap between well-off and low-income students increases at each grade level, for 26-27 through 28-29. Comment By comparing the changes at every score and grade level in Figure 13, one can construct the graph in Figure 14. In part (A) note that well-off students who have a 6% score at third grade flow to a score of 67% by eleventh grade, well above the eleventh grade score of low-income students who started at 9% at third grade. Part (B) shows the rate at which the gap between well-off and low-income students increases each year. The largest rates of increase occur at sixth and ninth grades.

16 4 TEXAS DATA Value-Added Measures for Schools School Value Added by Quartile.2.2 Probability.1 Probability.1. 15 1 5 5 1 Score Changes 26--27. 15 1 5 5 1 Score Changes 26--27 Poverty Concentration (Percent students in school eligible for free/reduced lunch) 1% % Figure 15: Schools are separated into four quartiles according to score gains in 9th grade mathematics, 24-26. These gains are then used to predict the score gains in 27. The lowest-quartile schools obtain on average a 5-point smaller gain than the highest-quartile schools. Poverty is more concentrated in the lowest-quartile schools than the highest-quartile schools. Comments Figure 15 separates schools into four quartiles depending upon the magnitude of student score gains in ninth-grade mathematics, averaging over 24-26. For each of the four quartiles, the probability of score changes in 26-27 is plotted as a result. The right hand plot compares lowest-quartile schools with highest-quartile schools, color coded for the concentration of poverty. The lowest-quartile schools have a higher concentration of poverty.

17 Value-Added Measures for Schools HighandLow-PerformingSchoolsinDalas 24-27 HighandLow-PerformingSchoolsinHouston, 24-27 27PerCapitaIncome to17,1 17,1to19. 19,9to22. 22,6to27. 27,5to14. NumbersofStudentsintaking9thgradeexam 3 5 1, 1,268 PerformanceStatus HighPerformers LowPerformers Figure 16: Lower-performing and higher-performing schools in Houston and Dallas. Comment Figure 16 shows the geographic distribution of schools in the Dallas and Houston areas, color coded according to whether score changes in ninth-grade mathematics were positive on average during 24-26 (higher-performing) or negative (lower-performing). The background colors indicate per capita income by zip code. Lower-performing schools are clustered together in low-income areas, with some exceptions.

18 4 TEXAS DATA School Size In small schools, student achievement increases, particularly for minority and low-income students. --- http://www.smallschoolsproject.org TexasSATCriterionversusSchoolSize(29) PovertyConcentration(Percent) 2 4 6 8 1 PercentGraduatesMeetingStateSATCriterion 8 6 4 2 K 2K 4K CampusSize K 2K 4K CampusSize K 2K 4K CampusSize K 2K 4K CampusSize K 2K 4K CampusSize K 2K 4K CampusSize PercentageMinorityStudents. 1. Figure 17: Graduates meeting SAT/ACT College Readiness Criterion [11 on SAT or 24 on ACT] versus school size and poverty concentration for (almost) all Texas high schools in 29. Data from the Texas Education Agency Academic Excellence Indicator System (AEIS). Comments For every level of poverty, increasing school size is correlated with increasing fractions of students meeting the criterion. The effect is strongest for low concentrations of poverty. For each concentration of poverty, there are some relatively high-scoring schools that are also small.

19 5 Charter Schools Charter schools give children the opportunity to achieve their dreams. Across America, parents are clamoring for charter schools boosting waiting lists at America s 5, charter schools to an all-time high, enough to fill another 5, charters. --Center for Education Reform Texas Graduates Meeting SAT Criterion (28) Percentage Meeting SAT Criterion 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Number of Graduates 1 1 5 Texas Charter School Graduates Meeting SAT Criterion (28) Percentage Meeting SAT Criterion 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Number of Graduates 1 1 5 Figure 18: Texas secondary charter schools and SAT college readiness criterion. All charter schools are colored in. Every disk is a school, and the area of the disk is proportional to the number of graduates. Comment: Almost all Texas charter high schools are considerably worse than comparable public schools by the SAT College-Readiness measure. Only one charter school stands out as comparable to the best public schools.

2 5 CHARTER SCHOOLS 1 Texas, Commended 11th Grade Math (29) 8 Commended (%) 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Taking 11 th Grade Math 1 1 5 Texas, Commended, Charters, 11th Grade Math (29) 1 8 Commended (%) 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Figure 19: Charter schools and commended mathematics scores at 11th grade. Taking 11 th Grade Math 1 1 5 Comment: When Commended status in eleventh-grade mathematics is used as the measure of performance, five secondary Texas charter schools (YES North, YES Southeast, Idea, Peak Preparatory High School, and KIPP) stand out with strong performance relative to public schools. Another 35 charter schools have performance comparable to public schools, and the remaining 1+ are worse.

21 Mathematics Score Growth in Regular Public and Charter Schools, 23-27 Previous score was... Score Change 1 9%--1% 1 2 Grade 9 Grade 1 Grade 11 Charter School Yes No 8%--9% Score Change 1 1 2 7%--8% Score Change 1 1 2 6%--7% Score Change 1 1 2 2 4 6 8 1 2 4 6 8 1 2 4 6 8 1 Previous score was... 5%--6% Score Change 4 2 Poverty Concentration Grade 9 Poverty Concentration Grade 1 Poverty Concentration Grade 11 Charter School Yes No 2 4%--5% Score Change 4 2 2 3%--4% Score Change 4 2 2 2%--3% Score Change 4 2 2 2 4 6 8 1 2 4 6 8 1 2 4 6 8 1 Poverty Concentration Poverty Concentration Poverty Concentration Figure 2: Texas high school charter schools and student gains. Score changes have been averaged over the years 23-27. Error bars on charter schools come from the sequence of annual score changes. The figure shows the mean score changes of students in all public schools with the same levels of poverty, compared with individual charter schools. Comment. For every level of starting performance and every concentration of poverty, the majority of secondary charter schools produces smaller student learning gains than the average result obtained by conventional public schools.

22 5 CHARTER SCHOOLS Texas Charter School Accountability 1 Texas, Passing 11th Grade Math (29) 8 Passing (%) 6 4 2 2 4 6 8 1 Non-Charter: Unacceptable Charter: AEA Acceptable Charter: Exemplary Charter: Recognized Charter: Acceptable Charter: Uncceptable Charter: AEA Unacceptable Taking 11 th Grade Math 1 1 5 Figure 21: Fractions of eleventh-grade students passing TAKS mathematics in 29 at all the Academically Unacceptable public schools (red) and all the Texas charter schools, with charter schools color coded according to their accountability system. Comment The majority of secondary charter schools are subject to the Alternative Education Accountability system, where the cutoffs for acceptable performance are much lower than in the Standard system. The schools colored blue are Acceptable according to Alternative Education Accountability, but most of them have mathematics performance worse than the public schools that are Academically Unacceptable under Standard Accountability.

23 Texas Charter School Accountability California Charter Schools California Students, Mean SAT, 28 2 Mean SAT Score 15 1 2 4 6 8 1 Minority Students (%) % 1% Number of Students 1 1 1 California Charter School Students, Mean SAT, 28 2 Mean SAT Score 15 1 2 4 6 8 1 Minority Students (%) % 1% Number of Students 1 1 1 Figure 22: Mean score of California seniors taking the SAT, color coded by the percentage of graduates who take it. Data from 28, source, California Department of Education. The lower panel colors in only charter schools Comments. California has one exceptional charter school, where all students are low-income, all take the SAT and all do well (Preuss School, UCSD). Fifteen other charter schools encourage large fractions of low-income students to take the SAT but their scores are low. The remaining, roughly 25 high schools with reported SAT averages are comparable to other public schools.

24 5 CHARTER SCHOOLS Florida Charter Schools Florida Charter Students, Level 5 Mathematics, 1th grade, 21 1 Mathematics Level 5 (%) 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% 1 Number of 1th Graders 1 1 5 Florida Charter Students, Level 5 Mathematics, 1th grade, 21 Mathematics Level 5 (%) 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Number of 1th Graders 1 1 5 Figure 23: Fractions of Florida 1th graders with score of 5 on the FCAT mathematics exam; level 3 is passing. Every disk is a school, and charter schools are highlighted in color. Comment: Florida has around 1 secondary charters with higher mathematics performance at tenth grade than comparable public schools. Most of the remainder are comparable to regular public schools, and around 3 are worse.

25 Texas Charter School Accountability New Jersey Charter Schools 1 New Jersey 11th Grade Advanced Proficient Math, 29 Advanced Proficient (%) 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Tested, 11th Grade 1 1 5 New Jersey Charters, Advanced Proficient Math, 29 1 Advanced Proficient (%) 8 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Tested, 11th Grade 1 1 5 Figure 24: Fractions of New Jersy 11th graders rated advanced proficient in mathematics at 11th grade. Every disk is a school, and charter schools are highlighted in color for the lower panel. Comment: New Jersey has around 11 secondary charters and only one of them can be compared with the better non-charter public schools for 11th grade mathematics

26 5 CHARTER SCHOOLS New York Charter Schools 1 New York Regent s Mathematics B, 29 8 Scoring 85-1 (%) 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Taking Regent s Math B 1 1 5 Charters, New York Regent s Mathematics B, 29 1 8 Scoring 85-1 (%) 6 4 2 2 4 6 8 1 Minority Students (%) % 1% Taking Regent s Math B 1 1 5 Figure 25: Percentage of New York high school students taking Regent s Mathematics B who score 85-1%. Every disk is a school, and charter schools are highlighted in color for the lower panel. Comment: Only 4 New York State charter schools have public data for Regent s Mathematics B, which indicates readiness for college-level mathematics.

27 Maxims Maxims 1. (Golden Rule) Provide the education for others children you would have for your own. 2. (Dobzhansky s Law) Nothing makes sense in public education except in light of poverty. 3. (Occam s Razor) When all other explanations fail, assume professionals are acting in their self-interest. 4. (Uncertainty Principle) Educational data tend to be most sparse where the interests of the country are most at stake.