Save the Children Literacy Programs: Results from the Comparative Pilot Study, 2009

POLICY STUDIES ASSOCIATES, INC. POLICY STUDIES Save the Children Literacy Programs: Results from the Comparative Pilot Study, 2009-10 Rebecca A. Romash Richard N. White Elizabeth R. Reisner December 2010 Prepared for Save the Children U.S. Programs 54 Wilton Road Westport, CT 06880

Contents Page Summary... 1 1. Overview of the Study... 4 2. Change in Performance on STAR Reading Assessments... 5 3. Measures of Involvement in AR Programming... 8 4. The Relationship among Gains in Reading Performance, Program Enrollment, and Involvement in AR Activities... 11 Appendix A: Using Normal Curve Equivalent Scores... A-1 Appendix B: Books Read and STAR Reading Assessment Performance, by Program... B-1

Save the Children Literacy Programs: Results from the Comparative Pilot Study, 2009-10 Summary This comparative pilot study of the Save the Children literacy program, an initiative intended to increase student literacy in high-need rural schools, found that students participating in the program achieved greater gains in reading performance on standardized assessments over the course of a school year than did similar nonparticipating students who attended the same schools. The difference in literacy gains between the two groups was statistically significant and substantial. Since the 2003-04 school year, Save the Children, U.S. Programs (SC), has supported programming designed to improve reading skills among struggling students in rural areas. Struggling students are defined as those who are not yet able to read and comprehend grade-level text. The literacy programs include both afterschool and in-school interventions with small groups of children. These are implemented by teachers and paraprofessionals and are designed to provide a range of activities aimed at increasing literacy achievement. Central to Save the Children s literacy programs is guided independent reading practice (GIRP), which provides regular opportunities to read just right books independently, under the guidance of a trained adult, and aims to increase children s motivation for reading and vocabulary growth. GIRP uses Accelerated Reader (AR), a reading management software program developed by Renaissance Learning, to monitor guided reading practice. Change in reading proficiency is measured using the STAR Reading assessment. In the afterschool setting, daily literacy activities occur within an hour-long Developing Readers (DR) literacy block. Primarily designed for struggling readers in grades 2-6, these activities include 30 minutes of GIRP, 20 minutes of fluency-building support, and a 10-minute read-aloud. The 20-minute fluency-building component stresses repeated reading in a supportive environment. Text is practiced aloud so that students pay attention to the phrasing, punctuation, and tone of the text. Unfamiliar vocabulary words are introduced, heard, and repeated daily in context and used in playing games, thus increasing a child s likelihood of mastering the vocabulary words. All these activities are intended to work in concert to increase the achievement of struggling readers. Save the Children services provided to the programs at the schools included in this study comprise the delivery of integrated in-school and afterschool literacy activities for children, including the following: Funding for literacy learning materials, including AR, Emergent Reader (ER) resources, small-group reading tutorial resources, library resources, computer equipment, and standardized testing (the STAR Reading assessment and the 1

STAR Early Literacy assessment) to permit periodic assessment of literacy proficiency Training of local staff in literacy instruction An integrated, web-based system for tracking information about participating children, the program services they receive, and their outcomes, including change in literacy growth Feedback and technical assistance based on program data collected This study was designed to compare the changes in performance on the STAR Reading assessment of students participating in the Save the Children literacy program during the 2009-10 school year to changes among nonparticipating students in the same schools. For this analysis, scores were obtained from 18 elementary schools across the country. These schools were selected because both participating and nonparticipating students used Renaissance Learning s Accelerated Reader and STAR Reading software during the school day and because there were sufficient numbers of students in the participant and nonparticipating groups (at least 20 students in each group) at all 18 schools. Participants received Save the Children Developing Reader programming as described previously, including GIRP, fluency building, and read-aloud, while nonparticipants did not. STAR Reading assessment scores and data on AR quiz performance were collected and analyzed for both groups of students. Data from the 2009-10 school year for 3,488 students in grades 2 6 (1,715 participants and 1,773 nonparticipants) were available for this analysis. Most participating and nonparticipating students demonstrated reading skills significantly below grade level at the start of the school year. Eighty-three percent of participants and 63 percent of nonparticipants scored below grade level on the first STAR Reading assessment administered during 2009-10. To control for differences between participants and nonparticipants in the distributions of reading proficiency demonstrated on the initial assessment, a statistical procedure known as propensity matching was employed to create a matched comparison group. This procedure identifies the group of nonparticipating students who most closely resemble the group of project participants on the measures of interest. For this study, the measures used in matching were each student s score on the initial STAR Reading assessment and grade level. The propensity procedure indentified matching groups of 1,249 participants and 1,249 nonparticipants. Key findings from the analysis included the following: As measured by the STAR Reading assessment, participants improved their literacy skills substantially over the course of the program period, moving closer to grade-level performance. The average change in scores among matched 2

participants was a gain of 6.5 normal curve equivalents (NCEs), 1 while matched nonparticipants averaged a gain of 0.6 NCEs, for a net difference of 5.9 NCEs. The difference in gains in assessment scores was statistically significant. The effect size of the differences in gains on the assessment was +0.29, which can be interpreted to mean that the differences are the equivalent of three months of additional schooling. Sixty percent of participants increased their literacy performance by at least 2 NCEs, which is considered to be a meaningful learning gain by Renaissance Learning, the publisher of the STAR Reading assessment. Among matched nonparticipants, 41 percent achieved the same threshold in literacy gains, a statistically significant difference. The proportion of participants reading at a level appropriate for their grade or above increased during the 2009-10 school year. On the initial STAR Reading assessment, 23 percent of both participants and matched nonparticipants read at or above grade level (50 NCEs or more). When the final STAR Reading assessment was administered, 33 percent of participants read at or above grade level, a net gain of 10 percentage points. Among matched nonparticipants, 24 percent scored at grade level or higher on their final assessment, an increase of 1 percentage point. Measures of engagement with the program s literacy activities also differed between participants and matched nonparticipants. On average, participants read 51 books during 2009-10, while nonparticipants read fewer books, an average of 37 books. Participants performed better on AR activities than did nonparticipants. Participants passed 93 percent of the quizzes they completed on the books they had read, while matched nonparticipants passed an average of 80 percent, a statistically significant difference. A regression model was developed to control for differences in students initial level of literacy proficiency and their level of participation in AR activities. The results showed that participation in the Save the Children literacy program was associated with gains in STAR Reading assessment scores. On average, after controlling for students initial level of test performance, grade level, and involvement with AR activities, participation in the literacy program was associated with a final score 3.8 NCEs higher than was achieved by similar nonparticipants. 1 Throughout this study, scores are reported in normal curve equivalents (NCEs). An NCE score is a standardized score (based on a normal distribution) in which scores represent equal intervals that range from 1 to 99. Unlike percentiles or grade equivalents, which are not equal interval scales, NCEs can be subtracted and averaged, allowing the computation of summary statistics of group performance. A more detailed explanation of NCE scores appears in Appendix A. 3

1. Overview of the Study The purpose of this study was to examine the impact of participation in a Save the Children Developing Reader literacy program on students literacy performance, as measured on a standardized reading assessment. The study compared the gains in literacy performance achieved by program participants to gains achieved by matched nonparticipating students in the same grades from the same schools. Both groups of students completed AR quizzes and STAR Reading assessments during the regular school day as part of the regular school curriculum. Save the Children literacy program participants either took part in GIRP during the school day, or participated in an afterschool program. For this study, the study team analyzed measures of student performance from 18 schools located in nine states from across the country. Analyses were conducted on a population of 1,249 participants and 1,249 nonparticipants who were matched in terms of grade level and performance on the first STAR Reading assessment of the school year. 2 Limiting the analysis on the matched subpopulations is a means to control for any selection bias in these characteristics of the students participating in Save the Children s literacy program. Minimizing this bias increases our confidence in interpreting any differences in performance at the end of the school year as associated with program participation rather than resulting from initial differences in the two groups of students. Nearly half of the students included in the analysis (49 percent) were enrolled in third or fourth grade. A breakdown of the study population by grade level is shown in Exhibit 1. 2009-10 Grade Level Exhibit 1 Study Population by Grade Level, in Percents Percent of Matched Participants Percent of Matched Nonparticipants Percent of Study Population 2 20 20 20 3 19 19 19 4 27 27 27 5 22 22 22 6 12 12 12 N 1,249 1,249 2,498 Exhibit reads: Twenty percent of both participants and nonparticipants included in this analysis were enrolled in second grade during 2009-10. On average, the students included in this analysis performed below grade level on their initial STAR Reading assessment. The average score was 34.4 NCEs, the equivalent of the 23 rd 2 A description of all of the participants and nonparticipants available for analysis and of the subgroup identified for analysis is presented in Appendix B to this report. 4

national percentile rank. This indicates that the typical student included in this study was reading at a level higher than 23 percent of students nationwide at the same grade level, but lower than 77 percent of students. Seventy-seven percent scored below 50 NCEs on their initial STAR Reading assessment, a common standard for defining below grade-level performance on a nationally normed assessment. 2. Change in Performance on STAR Reading Assessments For this study, change in a student s reading proficiency was computed by subtracting his/her first score on the STAR Reading assessment in the 2009-10 school year from his/her score on the last assessment completed during the 2009-10 school year. Analyses were limited to students who completed two STAR Reading assessments at least 90 days apart. The group of participants and nonparticipants were selected to have similar distributions of scores on the initial STAR Reading assessment, with both groups averaging 34.4 NCEs. However, participants achieved higher scores on their final assessment than did the matched nonparticipants (Exhibit 2). Participants averaged a score of 40.9 NCEs on their final assessment, while matched nonparticipants averaged 35.0 NCEs, a statistically significant difference. Exhibit 2 Measures of STAR Reading Assessment Performance Study Group Matched Participants (n=1,249) Matched Nonparticipants (n=1,249) Average Initial STAR Reading Assessment (NCEs) Average Final STAR Reading Assignment (NCEs) Average Change in NCEs 34.4 40.9 6.5 34.4 35.0 0.6 Exhibit reads: Both participants and matched nonparticipants averaged 34.4 NCEs on their initial STAR Reading assessment. The difference in the distributions of initial STAR Reading Assessment scores between the two groups was not statistically significant (p=.986). Between their first and last assessments, participants gained an average of 6.5 NCEs, and matched nonparticipants averaged an increase of 0.6 NCEs. The net difference in the change in performance was 5.9 NCEs, a statistically significant difference with an effect size of +0.29. A model for expressing the magnitude of change in scores on a nationally normed assessment, such as the STAR Reading assessment, has emerged from the literature on student performance on achievement tests. The findings consistently estimate that the effect size of the change in scores over a full school year is equivalent to one standard deviation, or an effect size of 1.0. If we assume that the change in student performance occurs evenly over a 10-month 5

school year, then each increment in effect size of 0.1 is the equivalent of the gain expected from one month of schooling. Among the participants in the literacy program during the 2009-10 school year, the effect size of the change in scores on the STAR Reading assessment, compared to those of nonparticipants matched on initial scores, was +0.29. According to this model, an effect size of this magnitude can be interpreted as indicating that participation in the Save the Children literacy program produced additional gains in literacy proficiency for the average participant that are the equivalent of the gains expected from three months of schooling. Proportion Making Meaningful Gains An issue in the analysis of change in scores on the STAR Reading assessment is determining how large a change must occur before it is considered to represent a meaningful gain in reading proficiency, after taking into account the measurement error inherent in any assessment tool. For the STAR Reading assessment, the minimum meaningful gain is 2 NCEs, according to the assessment publisher. Among the students participating in the literacy program, 60 percent achieved a gain of 2 NCEs or more, as did 41 percent of matched nonparticipants. The difference between these proportions was statistically significant (p<.05). Change in the Proportion of Students Scoring Above Grade Level As discussed above, a score of 50 NCEs on a nationally normed test is often interpreted as performing at grade level. Overall, 23 percent of Save the Children literacy program participants and matched nonparticipants scored at or above grade level on their initial STAR Reading assessment (Exhibit 5). 3 On the final STAR Reading assessment of the 2009-10 school year, 33 percent of participants performed at or above grade level, representing a net gain of 10 percentage points in the proportion meeting this standard. Among matched nonparticipants, 24 percent performed at or above grade level on their final assessment, representing a net gain of 1 percentage point. The difference in the proportions of students scoring at grade level or above on the final assessment was statistically significant (p<.05). Among the Save the Children literacy participants who scored below grade level on their initial STAR Reading assessment of 2009-10, 18 percent improved their scores enough to be classified as performing at or above grade level on their final assessment. Among the matched nonparticipants scoring below grade level on their initial assessment, 12 percent moved to grade level or higher during the same period. The difference in the proportion of students in the two groups moving from performing below grade level to at or above grade level was statistically significant. 3 The similarity in performance on the initial assessment is the intentional product of the propensity matching of the two groups of students. 6

Exhibit 5 Proportion of Participants and Matched Nonparticipants Scoring at Grade Level or Above 100 Percent of Students Scoring At or Above Grade Level on STAR Reading Assessment (50 NCEs) 80 60 40 20 23 23 24 33 0 Initial Assessment Matched Nonparticipants (N=1,249) Participants (N=1,249) Final Assessment Exhibit reads: Among both participants and matched nonparticipants, 23 percent of students scored at or above grade level (50 NCEs or higher) on their initial STAR Reading assessment. The magnitude of students gains on the STAR Reading assessment over the course of the 2009-10 school year was found to be negatively and significantly correlated with their level of performance on the initial STAR Reading assessment of the school year. The correlation was -0.29. This finding indicates that students beginning the year with the lowest assessment scores made the greatest gains. While it is likely that some portion of this correlation reflects regression to the mean between the two administrations of the assessment, the relationship is too strong and too widespread to be fully explained in this way. The correlation between initial performance and gains in scores was stronger, and more negative, among program participants than among nonparticipants (-0.34 and -0.26, respectively). Both correlations were statistically significant (p<.05). Because participants and nonparticipants were selected for analysis on the basis of their initial performance on the STAR Reading assessment, we interpret this difference as a preliminary indication that participation in the program is associated with gains in performance. Performance on STAR Reading Assessments by Grade Level An analysis of the average change in performance on the STAR Reading assessment by grade level shows that students in lower grades achieved larger gains, on average, than did 7

students in higher grades (Exhibit 7). In addition, program participants achieved larger gains than did matched nonparticipants at each grade level, a difference of 5 NCEs, on average. Among participants, the average change in assessment scores exceeded 6 NCEs in grades 2 5. Among nonparticipants, students in grades 2 4 achieved increases in performance, while those in grades 5 and 6 experienced decreases in performance. At every grade level, the participants in the literacy program had larger gains than did nonparticipants, and the differences were statistically significant (p<.05). Exhibit 7 Change in STAR Reading Assessment Scores (NCEs), by Grade Level 10 8.3 8 7.6 Change in STAR Reading Assessment Performance (NCEs) 6 4 2 0 2.5 2.8 1.6 6.4 6.2 2.8-2 -1.9-4 Second Grade Third Grade Fourth Grade Fifth Grade Sixth Grade Grade Level of Participants -3.4 Matched Nonparticipants (N=1,249) Participants (N=1,249) Exhibit reads: Among participants in the Save the Children literacy program who were enrolled in second grade, the average change in score on the STAR Reading assessment was a gain of 8.3 NCEs, while matched nonparticipants in the same grade averaged a gain of 2.5 NCEs. 3. Measures of Involvement in AR Programming The findings discussed above indicate that Save the Children literacy program participants achieved significantly greater gains in reading achievement, on average, over the course of the 2009-10 school year than did nonparticipating students. The theory of change for the literacy program indicates that increased participation in literacy activities, including GIRP using AR, is the primary mechanism through which a student s reading proficiency is improved. To determine whether the level of participation in AR activities differed among participants and nonparticipants, the research team examined the patterns of involvement in several AR program activities over the school year. 8

The AR program is designed to encourage students to read and understand books written at an appropriate level of difficulty for their reading skill level. After completing a book, the student takes a quiz on the content of the book. The results of the quiz are used to track changes in reading proficiency and to help identify additional books that are appropriate for the student s skill level. were: The study team examined six measures of participation in AR activities. These measures Number of books read Percent of quizzes passed Average percent of questions correct per quiz Average book level Percent of books read independently (not read to the student by an adult or read with an adult) Percent of books read that were nonfiction These measures were selected because of the expectation of a relationship between a student s level of involvement in each of these aspects of AR activities with his/her change in performance on the STAR Reading assessment. To verify our assumptions, the study team computed the correlation between each measure with each student s change in STAR Reading assessment performance, as shown in Exhibit 8. Exhibit 8 Correlation of Measures of AR Involvement with Change in STAR Reading Assessment Scores (NCEs) AR Involvement Measure Correlation Coefficient Statistically Significant? Number of books read 0.14 Yes Percent of quizzes passed 0.15 Yes Average percent of questions correct per quiz 0.20 Yes Average book level 0.19 Yes Percent of books read independently Percent of books read that were nonfiction 0.06 Yes 0.13 Yes Note: Data on AR involvement were not available for all students included in this study. AR data were available for 1,033 participants and 1,229 matched nonparticipants. Exhibit reads: The correlation between the number of books read over the school year and gains on the STAR Reading assessment was +0.14, a statistically significant relationship. Five of the measures were found to have positive and statistically significant correlations with the change in assessment scores, suggesting that the greater a student s involvement in these AR activities, the greater the increases in reading proficiency achieved. One measure, the average book level, showed a negative relationship, suggesting that students who read lower- 9

level books achieved larger gains. Since students who scored lowest on their STAR Reading assessment would be advised to read lower-level books, this finding is consistent with the earlier finding that students with lower initial assessment scores made larger gains between their initial and final assessment. Over the 2009-10 school year, participants in the Save the Children literacy program read more books than did nonparticipants. Participants read an average of 51 books and the median number of books they read was 40. Nonparticipating students read an average of 37 books over the course of the year, and the median number of books they read was 14. The difference in the number of books read was statistically significant (p<.05). This finding suggests that participation in the Save the Children literacy program is associated with greater involvement in AR activities. In grades 2 and 3, nonparticipating students read more books, on average, than did participating students (Exhibit 9), although the differences were not statistically significant. In grades 4, 5, and 6, this pattern was reversed. Participants read more books than did nonparticipants in the same grades. The differences in grades 4 6 were statistically significant (p<.05). Exhibit 9 Average Number of Books Read During the 2009-10 School Year, by Grade Level 100 80 79 71 64 66 Number of Books Read 2009-10 60 40 44 39 37 20 17 11 10 0 Second Grade Third Grade Fourth Grade * Fifth Grade * Sixth Grade * Participants Matched Nonparticipants Note: * Indicates that the difference in distributions between participants and matched nonparticipants was statistically significant. Exhibit reads: On average, Save the Children literacy program participants in second grade read 71 books during the 2009-10 school year, while matched nonparticipants read 79 books. 10

Literacy program participants performed substantially better on the quizzes taken for the books they read than did nonparticipants. Participants passed 93 percent of the quizzes they took, on average, while matched nonparticipants passed an average of 80 percent of quizzes. Grade-by-grade analysis shows that participants passed a higher proportion of quizzes across grades and that this difference was statistically significant at each grade level. Literacy program participants also correctly answered a higher proportion of the individual quiz items than did matched nonparticipants. Participants correctly answered an average of 87 percent of quiz items, while matched nonparticipants correctly answered 76 percent of items. Grade-by-grade analysis shows that this pattern occurred across grades and was statistically significant at every grade level. With respect to the book level, 4 nonparticipating students read books that were rated to be at a higher reading level than were books read by participants. The average book level for matched nonparticipants was 3.2, compared with 2.8 for participants. This pattern held true for students for grades 3 6 when the data were analyzed on a grade-by-grade basis. The differences were statistically significant (p<.05). As part of the AR program, students can read a book independently, read it with a teacher, or listen to the book being read aloud. Participants read 93 percent of the books they completed over the school year independently, while matched nonparticipants read 75 percent of their books independently. This pattern was observed at each grade level, and the differences were statistically significant. The AR program also tracks the books read by students according to whether they are fiction or nonfiction. A larger proportion of the books read by participants were nonfiction (35 percent) than were books read by matched nonparticipants (24 percent). This pattern was found at each grade level, and all differences were statistically significant. Based on these findings, the study team concludes that there are substantial differences in the level of involvement in the AR program between students who participated in the Save the Children literacy program and students who did not. 4. The Relationship Among Gains in Reading Performance, Program Enrollment, and Involvement in AR Activities As reported above, participants in the Save the Children literacy program achieved larger average gains on the STAR Reading assessment (6.5 NCEs) than did a group of nonparticipating 4 The book level represents the difficulty of the text of a specific book; the more difficult the text, the higher the book level. Students participating in the AR program are encouraged to select books at a level that corresponds to their specific reading proficiency, so that the book is neither too difficult nor too easy for them to read. 11

students in the same schools matched on their initial reading proficiency (0.6 NCEs). The difference is statistically significant and substantial. To determine how the difference in gains in reading proficiency between the two groups of students was moderated by each student s level of involvement in AR activities, the research team conducted a multiple regression analysis taking into account seven independent variables: Whether a student participated in the Save the Children literacy program The student s grade level A student s score (NCEs) on the first STAR Reading assessment of the year (to control for prior performance) The number of books a student read over the course of the year (to control for involvement in AR literacy programming) The proportion of quiz questions answered correctly The proportion of nonfiction books read The average level of the books read by the student over the 2009-10 school year Standardized and unstandardized multiple regression coefficients for each variable are shown in Exhibit 10. The final model had an R 2 of 0.67, indicating that the model explained two-thirds (67 percent) of the variation observed in the final STAR Reading assessment scores of participants and nonparticipants in this study. The remaining variation is most likely explained by factors not measured for this study, such as student characteristics not included in the model (e.g., motivation, attachment to school), characteristics and effectiveness of teachers and schools, and parental attitudes toward school and involvement in their child s education. The results of multiple regression analysis show that participation in the Save the Children literacy program has a statistically significant, positive relationship with a student s performance on the final STAR Reading assessment of the 2009-10 school year, even after controlling for a student s performance on the initial assessment in the fall and involvement in the AR program. Overall, participation in the program was associated with an additional 3.8 NCEs on a student s final STAR Reading assessment score. This significant association of participation with higher final STAR Reading assessment scores holds true for students in every grade when the data are analyzed on a grade-by-grade basis. The regression analysis also shows that the number of books a student read, the percent of quiz questions answered correctly, the percent of books read that were nonfiction, the average level of the books read, and a student s score on the first STAR Reading assessment were each associated with higher scores on the final assessment when all other variables in the model are held constant. Only student grade level had a negative association, meaning that students in lower grades achieved larger gains in assessment scores than did those in higher grade levels. On the basis of these data and analyses, the study team concludes that participation in the Save the Children literacy program was associated with statistically significant, positive gains in reading proficiency when compared with changes in reading proficiency among similar nonparticipating students in the same schools. Furthermore, having controlled for the level of 12

involvement in AR activities and the differences in the level of initial reading proficiency in the final model, this study concludes that the reading program components present in the Save the Children literacy program provided a noteworthy increment to the gains in reading proficiency among the students at the participating schools, and that this difference was both statistically significant and substantive. Exhibit 10 Regression Model for Predicting Students Final STAR Reading Assessment Scores Final STAR Reading Assessment Score (NCEs) = + Whether participated in literacy program * 3.80 + Initial STAR Reading Assessment (NCEs) * 0.69 + Number of books read in 2009-10 * 0.01 + Percent of quiz questions answered correctly * 21.52 + Proportion of books read that were nonfiction * 2.19 + Average level of books read * 2.98 - Student grade level * 2.22-6.61 (the model constant) Predictor Variable Unstandardized Coefficient Standardized Coefficient (β) Significance (p) Participation in literacy program 3.80 0.09 0.00 Number of books read 0.01 0.04 0.04 Percent of quiz questions 21.52 0.17 0.00 answered correctly Percent of books read that were 2.19 0.03 0.03 nonfiction Average level of books read 2.98 0,13 0.00 Student grade level -2.22-0.14 0.00 Score on first STAR assessment 0.69 0.68 0.00 (NCEs) Constant -6.61 0.00 Note: Analysis was conducted on participants and matched nonparticipants (1,249 in each group). Exhibit reads: Participation in the literacy program was associated with a score on the final STAR Reading assessment that was 3.80 NCEs higher than that of students who did not participate in the program, after controlling for all other variables in the model. 13

Appendix A Using Normal Curve Equivalent Scores

Appendix A Using Normal Curve Equivalent Scores A normal curve equivalent (NCE) score is a standardized score (based on a normal distribution) that makes it possible to compare scores across grades and to compute statistics that describe the performance of a group of children, such as average scores, that are not possible using other ways of expressing test scores. NCEs consist of 99 equal units, an advantage over percentiles and grade equivalents in analyzing changes in test scores. Because NCE scores are equal units, they can be averaged for a group of children. NCEs are computed by dividing the normal curve distribution of student scores on an assessment into 99 units, with each NCE unit spanning the same number of test points. NCE scores have a mean of 50 and a standard deviation of 21.06. In a normal distribution, an NCE of 1.0 is equivalent to a percentile rank of 1.0, and an NCE of 99.0 is equivalent to a percentile rank of 99.0. Percentile ranks, as can be seen in Exhibit A1, tend to cluster around the middle of the distribution of test scores, so that a change in performance of one percentile represents a smaller change in the raw score on an assessment at the center of the distribution than at the extremes of the distribution. Because NCEs comprise 99 units of equal size, a change of one NCE represents the same change in the raw score at any point in the distribution. Exhibit A1 Distribution of Student Scores, Normal Curve Equivalent Scores, and Percentiles Normal Curve Equivalent Percentile Children performing at grade level will have an NCE of 50, regardless of their grade in school. If a student makes exactly one year of progress after one year of instruction, his/her NCE score will be exactly the same for both years, and the change in NCE scores will be zero. A student with a one-year change in NCE scores that is greater than zero has achieved larger gains than the general population, and a student with a negative change in NCE scores has made less progress. A-1

Appendix B Characteristics of the Population of Participants and Nonparticipants Available for Propensity Matching

2009-10 Grade Level Exhibit B1 Total Student Population of the 18 Schools, by Grade Level, in Percents Percent of All Participants Percent of All Nonparticipants Percent of Total Population 2 20 20 20 3 24 19 22 4 28 22 25 5 19 21 20 6 9 18 14 N 1,715 1,773 3,488 Exhibit reads: Twenty percent of all participants and of all nonparticipants were enrolled in second grade during 2009-10. The difference in the distribution of students by grade level was statistically significant (p<.05). Exhibit B2 Performance on Initial STAR Reading Assessment, All Participants and Nonparticipants Initial STAR Reading Assessment Score (NCEs) Participants (N=1,715) Nonparticipants (N=1,773) Average Initial Score 31 NCEs 39 NCEs Exhibit reads: Among all Save the Children literacy program participants, the average score on the initial STAR Reading assessment was 31 NCEs. The difference in the distribution of scores between participants and nonparticipants was statistically significant (p<.05). B-1

Exhibit B3 Distribution of Scores on Initial STAR Reading Assessment, All Participants and Nonparticipants Initial STAR Reading Assessment Score (NCEs) Percent of All Participants Percent of All Nonparticipants <10 15 14 10-19 15 7 20-29 17 11 30-39 20 15 40-49 16 16 50-59 9 16 60-69 5 12 70-79 2 6 80-89 1 2 90+ 0 1 N 1,715 1,773 Exhibit reads: Fifteen percent of all Save the Children literacy program participants scored below 10 NCEs on their initial STAR Reading assessment. The difference in the distribution of scores between participants and nonparticipants was statistically significant (p<.05). B-2