Good Habits, Great Readers Program

Transcription

1 A Final Report for the Evaluation of Pearson s Good Habits, Great Readers Program November 18, 2010 Evaluation of Pearson s Good Habits, Great Readers Program i

2 EXECUTIVE SUMMARY Educators share a common goal of maximizing student learning and achievement, and they understand the importance of effective, research-based curricula in helping them achieve this goal. Educators also understand that students have diverse learning needs and that tailoring instruction to specific needs and skills is critical to ensuring students academic success. Therefore, among today s educators, there is a high demand for comprehensive literacy programs that facilitate differentiated instruction and enable educators to meet the needs of all students. Pearson s Good Habits, Great Readers program uses a comprehensive, balanced approach to literacy instruction. The program includes whole-group reading instruction (shared reading) and small-group instruction (guided reading) to encourage students reading independence. The shared reading program components provide teachers with an opportunity to introduce important strategies and skills, and the guided reading program components provide support to teachers in differentiating their reading instruction based on each student s specific level. Pearson appreciates the importance of establishing the effectiveness of its programs through research and evaluation. Therefore, it has contracted with Magnolia Consulting, LLC, an external, independent consulting firm specializing in educational research and evaluation, to conduct an independent efficacy study of its Good Habits, Great Readers program. STUDY DESIGN & METHODS Magnolia Consulting conducted an independent quasi-experimental evaluation study of Pearson s Good Habits, Great Readers program with fourth- and fifth-grade students over the school year. The study s final sample included 44 teachers and 673 students across 12 geographically diverse schools. The purpose of this efficacy study was to evaluate teachers perception and implementation of the Good Habits, Great Readers program and to assess the effectiveness of the materials in helping students attain essential reading and writing skills. This study also compared the reading and writing performance of students who participated in the Good Habits, Great Readers program to students participating in comparison reading programs. This study also examined student attitudes in reading among Good Habits, Great Readers participants and among students who participated in comparison programs. Evaluators used a mixed-methods design to collect qualitative and quantitative data throughout the study period. The measures included pretest and posttest student assessments (Gates-MacGinitie Reading Test, 4th edition (GMRT-4),Developmental Reading Assessment K 3, Second Edition (DRA2)), a survey to gauge reading attitudes, implementation logs, a teacher survey, classroom observations, and interviews. GOOD HABITS, GREAT READERS PROGRAM IMPLEMENTATION Key Question: How do teachers implement the Good Habits, Great Readers program? In this study, most treatment teachers implemented the Good Habits, Great Readers program with high fidelity (84%), and the remaining teachers implemented the program with moderate fidelity (16%) in comparison to the implementation guidelines. Therefore, teachers generally implemented the program appropriately. Over the school year, teachers spent an average of 3.83 days using the Reading components of the program and an average of 3.55 days using the Writing program components. On average, teachers used small Evaluation of Pearson s Good Habits, Great Readers Program ii

3 groups for guided reading at least three days per week. Key Question: What are teachers perceptions of the quality and utility of the Good Habits, Great Readers program? Teachers who implemented Good Habits, Great Readers over the study period shared mixed feedback about the program. Many teachers appreciated the comprehensive nature of the Good Habits, Great Readers reading lesson plans. Several teachers also shared their appreciation for the Good Habits, Great Readers Writing program. Across treatment sites, administrators agreed that they valued the balanced literacy approach embedded in Good Habits, Great Readers, although several administrators also noted that no single program could meet all their needs. In general, teachers thought the Good Habits, Great Readers program plenty of material to cover, with only 11% of teachers reporting that the program did not provide enough material. In many cases, teachers reported supplementing the program, often to meet students needs or to help students prepare for state testing. Most teachers indicated that the program pacing allowed them to somewhat address or fully address the needs of all students. Several teachers praised specific components of the Good Habits, Great Readers program, including the best practice routines, the guided reading materials, and the consumable Grammar Usage and Mechanics books. A few teachers also connected students experiences in guided reading to writing, indicating that the guided reading topics often motivated their students and that their students wrote more with this program than they had in the past. Teachers also shared feedback about ways in which they thought the program was lacking. In particular, some teachers thought the shared reading program was too repetitive, and some noted a lack of alignment between the embedded assessments and the state standardized tests. A few teachers also expressed that they would like to see more from a writing program. Teachers often reported that most of their students displayed average or high engagement in the Good Habits, Great Readers materials. Furthermore, most teachers agreed that the Good Habits, Great Readers materials were at least somewhat adequate in meeting the needs of belowlevel, on-level, and advanced readers. Observation and interview data revealed that compared to the teachers implementing comparison programs, treatment teachers spent more time building background knowledge, addressing context, and engaging in explicit skill instruction during shared reading lessons. Treatment teachers also spent more time differentiating instruction than did comparison teachers. Another notable difference between treatment and comparison classrooms was that teachers using Good Habits, Great Readers had comprehensive guides to support their shared reading, guided reading, and writing instruction, and most comparison teachers did not have these explicit support materials. STUDENT LEARNING AND ATTITUDE RESULTS Key Question: Do students who participate in the Good Habits, Great Readers program demonstrate learning gains in reading during the study period? On average, students participating in Good Habits, Great Readers demonstrated gains on the GMRT-4 Vocabulary, Comprehension, and Total tests. Furthermore, the gains on the Vocabulary and Total tests were significant, corresponding to large (d = 0.56) and medium (d = 0.37) effect sizes, respectively. Gains on the Comprehension test were not significant and corresponded to a small effect size (d = 0.13).On average, Good Habits, Great Readers program participants learning gains reflect percentile gains of 21 points on the Vocabulary test, 5 points on the Evaluation of Pearson s Good Habits, Great Readers Program iii

4 Comprehension test, and 14 points on the Total test Pretest Vocabulary Comprehension Posttest In addition to demonstrating gains on the GMRT-4, on average, fourth- and fifth-grade students who participated in the Good Habits, Great Readers program showed steady increases in their DRA2 text-level scores over the study period, and their mean end-of-year text-level scores were consistent with grade-level benchmarks. These results indicate that participating in Good Habits, Great Readers throughout the study period was associated with learning gains among fourth- and fifth-grade students. Key Question: Is participation in the Good Habits, Great Readers program associated with differential reading gains for different subgroups of students? Subgroup analyses revealed some significant differences by subgroup, including limited English proficiency status and special education status, suggesting that the program might work differently for certain subgroups of students. Analyses examining gains across student subgroups based on initial reading performance revealed no significant differences across groups, suggesting that the Good Habits, Great Readers program might work equally well, regardless of pretest reading level. In addition, the results from the subgroup analyses by teacher characteristics suggest that the Good Habits, Great Readers program might work equally well for students, regardless of various teacher characteristics examined. Key Question: Do students who participate in the Good Habits, Great Readers program demonstrate changes in attitudes toward reading over the study period? On average, Good Habits, Great Readers program participants demonstrated significant decreases in Recreational Reading Attitude over the study that corresponded to a small effect size (d = 0.26). Changes in Academic Reading Attitude were not significant among Good Habits, Great Readers participants and corresponded to a small effect size (d = 0.10). Although treatment students did not demonstrate positive changes in reading attitude, their average attitude toward reading remained on the positive end of the response scale throughout the study period. Key Question: How does the reading performance of students who participate in the Good Habits, Great Readers program compare to that of students participating in other reading programs? On average, there were no significant differences in learning gains among Good Habits, Great Readers participants and comparison-group participants. Although treatment students gained less, on average, than comparison-group participants on the GMRT-4 Vocabulary, Comprehension, and Total tests, none of the differences was significant, and they corresponded to small effect sizes (d = 0.20, d = 0.21, and d = 0.24, respectively). These findings suggest that students who participated in Good Habits, Great Readers demonstrated gains on the GMRT-4 comparable to those of students who participated in comparison programs during the study period. Therefore, whether a student participated in Good Habits, Great Readers or in a comparison program did not have a significant impact on learning gains measured by the GMRT-4. Evaluation of Pearson s Good Habits, Great Readers Program iv

5 Most students in the treatment and comparison groups read at or above benchmark DRA2 reading stages by the end of the study period. However, compared to comparison-group students, Good Habits, Great Readers participants demonstrated a greater increase in the percentage of students who moved from reading below benchmark DRA2 reading stages at the beginning of the study to reading at or above benchmark DRA2 reading stages at the end of the study period. Analyses examining subgroup effects by study condition suggest that comparison programs were more effective than Good Habits, Great Readers for females, but Good Habits, Great Readers was more effective than comparison programs for students with limited English proficiency and for special education students. This finding suggests that the Good Habits, Great Readers program might be effective at reducing the achievement gaps between students with limited English proficiency and those proficient in English and between special education students and non-special education students. Key Questions: How do reading attitudes of students participating in the Good Habits, Great Readers program compare to those of students participating in other programs? On average, Good Habits, Great Readers program participants gained less than comparison program participants did in Recreational Reading Attitude, Academic Reading Attitude, and Total Reading Attitude. These differences were not significant and corresponded to small but notable effect sizes (d = 0.19 for Recreational Reading Attitude, d = 0.26 for Academic Reading Attitude, and d = 0.26 for Total Reading Attitude). These findings suggest that, on average, whether a student participated in Good Habits, Great Readers versus comparison programs did not significantly affect changes in reading attitude over the study period. Key Question: How does the end-of-year writing performance of students participating in the Good Habits, Great Readers program compare to that of students participating in other programs? Analyses examining end-of-year writing performance among treatment and comparison groups indicated that although there were no significant differences between Good Habits, Great Readers participants and comparisonprogram participants, there were notable effect sizes for some writing subtests. In particular, the effect sizes for Writing Ideas and Writing Word Choice were above 0.20 and favored treatmentgroup students (d = 0.26 and d = 0.23, respectively), and the effect size for Writing Convention was notable and favored comparison-group students (d = 0.26). These findings suggest that Good Habits, Great Readers participants and comparison-group participants demonstrated similar end-of-year writing performance. Overall, this study revealed that teachers liked a number of Good Habits, Great Readers attributes, but they also found areas where the program did not meet all their needs. Administrators acknowledged that it would be highly unlikely for one program to meet everyone s needs and concurred that, in general, Good Habits, Great Readers met their needs. Regarding student learning, on average, fourth- and fifth-grade students participating in the Good Habits, Great Readers program over the school year demonstrated learning gains in reading, some of which were significant. Moreover, their learning gains were comparable to those of students participating in other high-quality reading programs. Evaluation of Pearson s Good Habits, Great Readers Program v

6 ACKNOWLEDGEMENTS This evaluation study of the Good Habits, Great Readers program was a collaborative effort among Magnolia Consulting, Pearson, and study participants across 12 schools. Magnolia Consulting gratefully acknowledges the individuals who contributed their time and efforts to make this study a success. We would especially like to thank the participating teachers, site coordinators, administrators, and students who participated in the study. Magnolia Consulting also offers our sincerest thanks to the staff at Pearson who understand the importance of independent research studies as well as the critical components of rigorous applied evaluations. Finally, we would like to express our appreciation to Dr. Stephanie Baird Wilkerson, Dr. Jennifer Watts, Arianne Welker, Candace Rowland, and the entire Magnolia Consulting team. The authors, Lisa C. Shannon, PhD Mary K. Styers, PhD Evaluation of Pearson s Good Habits, Great Readers Program vi

7 TABLE OF CONTENTS EXECUTIVE SUMMARY... i ACKNOWLEDGEMENTS... vi INTRODUCTION... 1 RESEARCH DESIGN... 2 METHODOLOGICAL APPROACH... 3 MEASURES... 4 PROCEDURES... 7 SETTINGS PARTICIPANTS PROGRAM DESCRIPTION GOOD HABITS, GREAT READERS (GHGR) COMPARISON PROGRAMS PROGRAM IMPLEMENTATION IMPLEMENTATION BY TREATMENT TEACHERS IMPLEMENTATION BY COMPARISON TEACHERS STUDENT PERFORMANCE AND ATTITUDE RESULTS LEARNING GAINS AMONG GOOD HABITS, GREAT READERS PARTICIPANTS CHANGES IN READING ATTITUDES AMONG GHGR PROGRAM PARTICIPANTS STUDENT LEARNING COMPARISONS IN FOR READING PERFORMANCE COMPARISONS IN STUDENT READING ATTITUDES STUDENT END-OF-YEAR WRITING COMPARISONS SUMMARY AND DISCUSSION REFERENCES APPENDICES Evaluation of Pearson s Good Habits, Great Readers Program vii

8 INTRODUCTION Most educators understand that learning how to read effectively is one of the most important skills for children to master in the elementary school years. Being a strong reader provides children with a means to acquire knowledge, which is critical not only to school success, but to many other aspects of life as well. As researchers continue to understand ways to support students in becoming skilled readers, many find that using a comprehensive, balanced approach to literacy that combines wholeclass and small-group instruction is one effective approach. This balanced approach to literacy instruction posits that not only is the content of the instruction important, but that the methods of instruction are also critical (National Institute of Child Health and Human Development (NICHD), 2001; Neuman and Dickinson, 2001). One area that has become the focus of many research and evaluation efforts in recent years is reading comprehension. Proficiency in reading comprehension is essential for students to gain meaning from the material they read. The National Reading Panel addresses critical issues regarding reading comprehension in a comprehensive report and posits that effective reading comprehension programs should do the following: a) help readers gain insight into important metacognitive processes associated with reading comprehension instruction and learning; 2) facilitate students reading comprehension by modeling and sharing effective reading strategies; and 3) support readers appropriately, so they can practice reading strategies with teacher guidance until they internalize them and can apply them independently (NICHD, 2001). These recommendations highlight the importance of teaching metacognitive processes associated with effective reading, modeling and sharing effective reading strategies, and providing appropriate levels of scaffolding. Pearson appreciates the importance of these critical components, and they developed a comprehensive reading program, Celebration Press Reading: Good Habits, Great Readers 2007 (Good Habits, Great Readers), to facilitate the development of effective teachers and strong readers. Good Habits, Great Readers addresses critical components for nurturing effective reading comprehension skills by providing instructional methods and materials that encourage a gradual release of responsibility from teachers to students (Frey, 2006). The Good Habits, Great Readers program aligns to this model by supporting reading instruction that proceeds gradually from teacher modeling to student partner practice to independent application of skills and strategies. The Good Habits, Great Readers program uses a comprehensive, balanced approach that includes whole-group reading instruction (shared reading) and small-group instruction (guided reading) to encourage students reading independence. In the Good Habits, Great Readers program, shared reading provides an opportunity for teachers to introduce important strategies and skills, and guided reading provides teachers with the opportunity to provide more targeted, customized reading instruction based on each student s specific level. Good Habits, Great Readers also provides literacy centers, which provide students with regular opportunities to engage in independent and small-group activities to apply and reinforce what they learn (Pearson Education, Inc., 2007). One overarching concept behind the Good Habits, Great Readers program is that good readers use effective strategies (good habits) and that teachers can facilitate students learning and use of such strategies in their reading. Therefore, an important component of the program is that it provides explicit models for instruction in reading strategies and skills that help students develop Evaluation of Pearson s Good Habits, Great Readers Program 1

9 metacognitive strategies that help them actively make sense of the text they read (Pearson Education, Inc., 2007). Pearson supported a previous evaluation investigating the effectiveness of the Good Habits, Great Readers program among first-grade students. The findings indicated that first-grade students who participated in the Good Habits, Great Readers program experienced significant learning gains in reading proficiency, and the gains were similar to those of students who participated in other highquality reading programs (Shannon, Wilkerson, & Herman, 2007). As part of their ongoing process to demonstrate the effectiveness of their curriculum materials, Pearson has contracted with Magnolia Consulting, LLC, an external, independent consulting firm specializing in educational research and evaluation, to conduct an independent efficacy study of its Good Habits, Great Readers reading curriculum materials for fourth- and fifth-grade students. Magnolia Consulting conducted this study for Pearson throughout the school year. This final report presents an overview of the study s research design and methods, a description of the Good Habits, Great Readers program and the comparison programs used during the study, an examination of program implementation, results regarding student performance on various assessments, and a discussion of the study s findings. Evaluation of Pearson s Good Habits, Great Readers Program 2

10 RESEARCH DESIGN The purpose of this efficacy study was to evaluate teachers perception and implementation of the Good Habits, Great Readers program and to assess the effectiveness of the materials in helping students attain essential reading and writing skills. This study also compared the reading and writing performance of students who participated in the Good Habits, Great Readers program to students participating in competing comprehensive reading programs. In addition, the study examined whether students who participated in the Good Habits, Great Readers program demonstrated changes in reading attitudes over the study and examined how they compared to those of comparisonprogram participants. Magnolia Consulting evaluators conducted the study with fourth- and fifthgrade participants across 12 schools. The study addressed the following overarching evaluation questions: 1. How do teachers implement the Good Habits, Great Readers program? 2. What are teachers perceptions of the quality and utility of the Good Habits, Great Readers program? 3. Do students who participate in the Good Habits, Great Readers program demonstrate gains in reading over the study? 4. Is participation in the Good Habits, Great Readers program associated with differential reading gains for different subgroups of students? 5. Do students who participate in the Good Habits, Great Readers program demonstrate changes in reading attitudes over the study? How does reading attitude relate to reaching achievement among Good Habits, Great Readers participants? 6. How does the reading performance of students who participate in the Good Habits, Great Readers program compare to that of students who participate in other reading programs? 7. How do reading attitudes among students who participate in the Good Habits, Great Readers program compare to those of students who participate in other reading programs? 8. How does the writing performance of students who participate in the Good Habits, Great Readers program compare to that of students who participate in other writing programs at the conclusion of the study? METHODOLOGICAL APPROACH This study used a quasi-experimental design. Evaluators use quasi-experimental designs often in educational research when random assignment is not feasible. The treatment group consisted of participants from schools currently using the Good Habits, Great Readers program and the comparison group consisted of participants from schools using other reading programs. Evaluators chose this design at Pearson s request to examine student performance within schools that had experience implementing the Good Habits, Great Readers program. In this case, random assignment to the experimental groups was, therefore, not possible. To strengthen the design, evaluators tried to recruit treatment- and comparison-group schools that were comparable regarding their size and demographics. Because it was not always possible to obtain a strong match, evaluators conducted pretest analyses to examine pretreatment equivalence and included appropriate covariates in Evaluation of Pearson s Good Habits, Great Readers Program 3

11 subsequent analyses to control for any preexisting group differences. However, the lack of random assignment to these groups could have still introduced a selection bias in which any prior differences between the groups could have contributed to study outcomes. Evaluators strengthened this design by conducting pre and post observations of student learning on multiple outcome measures. In this study, students were nested in classrooms in which teachers implemented the treatment or comparison reading programs. Due to this nested data structure, evaluators used multilevel modeling to analyze these data. Using multilevel modeling enabled evaluators to control for student-level and classroom-level sources of variance that could affect student outcomes. 1 In designs in which students are nested in classrooms, it is important to account for these different levels of variance because students within the same classroom also share many experiences associated with those classroom factors. Therefore, they might respond more similarly to a particular reading program than students in a different classroom (Borman et al., 2005). Multilevel modeling is an appropriate analytic technique to account for the interdependence among the data. In addition to using multilevel modeling to analyze student performance outcomes, evaluators also calculated effect sizes to determine the magnitude of changes in treatment students assessment scores and to determine the magnitude of differences in assessment scores between treatment- and comparison-group students. 2 Evaluators also used descriptive and nonparametric analytic techniques to provide information regarding participant characteristics, teacher program implementation, and changes in students learning and reading attitudes. Finally, evaluators conducted analyses to examine the comparability of treatment and comparison groups and to examine attrition rates throughout the study period. MEASURES In the conduct of this study, evaluators used a mixed-method design. This design used qualitative and quantitative methodology to (a) explore how teachers implemented the Good Habits, Great Readers program and comparison programs; (b) assess teachers experience with and perceptions of the programs they used; (c) determine whether participating in the Good Habits, Great Readers program was associated with improvements in reading skills and interest; and (d) compare the effectiveness of Good Habits, Great Readers to the effectiveness of comparison programs. This portion of the report provides an overview of the student and teacher measures used to help the study achieve its goals. Student Measures Evaluators used the Gates-MacGinitie Reading Test (GMRT-4), and the Developmental Reading Assessment K 3, Second Edition (DRA2), to assess students reading skills throughout the 1 Student-level variance can be associated with factors such as student demographics and pretest reading ability. Classroom-level variance can be associated with factors such as teacher training and experience and reading programs. 2 An effect size is a unit of measurement that expresses the difference in outcome for the average treatment participant from the average control student. Effect sizes indicate the strength of the increase or decrease in achievement of students in the treatment group. Effect sizes are expressed in standard deviation units. For example, an effect size of 1.0 would indicate that the mean score in the treatment group was 1.0 standard deviation higher than the mean score in the control group. Evaluation of Pearson s Good Habits, Great Readers Program 4

12 program. Evaluators also used the Elementary Reading Attitude Survey (ERAS) (McKenna & Kear, 1990) to assess students academic reading attitude, recreational reading attitude, and overall attitude toward reading. These assessments were appropriate to gauge the outcomes of attitude and demonstrate high reliability (see Appendix C for reliability information). Teachers administered the above assessments to all treatment and comparison students who participated in the study. In addition, a sample of students from treatment and comparison classrooms completed a student writing assessment at the end of the study. Gates-MacGinitie Reading Test (GMRT-4) The GMRT-4 is a standardized norm-referenced assessment that includes measures of vocabulary and reading comprehension. The test is group-administered, using paper and pencil, and takes approximately 55 minutes to complete. Two forms, S and T, allow for pretesting and posttesting over the course of a study. Riverside Publishing services provide separate scores for Vocabulary, Comprehension, and Total, including raw scores, scale scores, percentile ranks, stanines, normal curve equivalents, and grade equivalents. Developmental Reading Assessment, Second Edition (DRA2) The DRA2 is an individually administered assessment that measures student performance in the following areas: (a) reading engagement, (b) oral reading fluency, and (c) comprehension. This assessment is fully integrated into the Good Habits, Great Readers program. The DRA2 uses benchmark books to assess student performance in reading and enables teachers to match readers to leveled reading books appropriate for their level of proficiency. Administration time is approximately 20 to 40 minutes per student, and teachers administered the DRA2 at the beginning, middle, and end of the year. Teachers score the assessment, and it yields scores for a reading level, reading engagement, oral reading fluency, and comprehension. Student Writing Assessment To obtain a sample of student narratives, evaluators asked sites to collect a sample of writing assessments from approximately one third of all fourth- and fifth-grade treatment students and one third of fourth- and fifth-grade comparison students at the end of the study. Evaluators used narrative writing prompts from state assessments in Arkansas and Maine. An evaluator with previous classroom experience graded all the writing assessments using the Good Habits, Great Readers short story rubric within the section on rubrics for Anchor Papers. This rubric allowed for a set of six writing scores on a 1 4 scale, including ideas, organization, voice, word choice, sentence fluency, and conventions. Additionally, evaluators created a total writing score. Appendix C provides a description of the writing prompts and scoring procedure. Elementary Reading Attitude Survey (ERAS) (McKenna & Kear, 1990) The ERAS (McKenna & Kear, 1990) is a 20-item survey assessing students level of academic, recreational, and total reading attitude. Each of the twenty items is scored on a four-point scale, and it visually depicts various levels of emotion using Garfield pictures. Teachers administered this survey to their classrooms by reading items aloud and asking students to respond with how they feel about each statement on their own answer sheets. Administration time was approximately minutes. Teachers returned the surveys to Magnolia Consulting for scoring. Surveys are coded on a 4-point scale, ranging from 1, feeling very upset, to 4, feeling very happy. The survey yielded three raw scores for academic reading attitude, recreational reading attitude, and total reading attitude. Evaluators calculated mean scores for each category and used these scores in the main analyses. Evaluation of Pearson s Good Habits, Great Readers Program 5

13 Teacher Measures Evaluators used a number of data collection techniques to assess program implementation as well as teachers perceptions of their reading programs. Treatment teachers participated in spring classroom observations and interviews. They also completed online implementation logs for each of the seven units. Comparison teachers participated in spring classroom observations and interviews. The resulting data yielded valuable information regarding reading and writing instruction and its efficacy in treatment and comparison classrooms. Using multiple teacher measures allowed evaluators to triangulate data through multiple methods, collect data throughout the project, and obtain data from all participants, all of which strengthened the validity of the study s results (Erickson, 1986). Classroom Observation Protocol To increase the validity and reliability of classroom observations, evaluators developed detailed observation protocols around five major constructs: 1. the classroom environment 2. teachers instructional practices 3. shared reading instruction 4. guided reading instruction 5. writing instruction Evaluators developed a separate protocol for treatment and comparison classrooms, and although the protocols were similar and reflected best practices in reading instruction (Lyons & Pinnell, 2001; Vaughn & Linan-Thompson, 2004), the treatment classroom protocol included items specific to the Good Habits, Great Readers program. Evaluators structured the protocols so that each main construct was subdivided into smaller constructs that included checklist items and areas to write observational notes about exactly what was observed. The classroom environment construct focused on the availability and nature of reading and writing materials, space and room arrangement, and displays. The instructional practices construct focused on use of groups, the use of assessment, and teacherstudent interactions. The shared reading instruction construct focused on the lesson introduction, main lesson, and further instruction. The guided reading instruction construct focused on before, during, and after reading lessons, as well as on further instruction. Finally, the writing instruction construct focused on a typical writing lesson, trait time lessons, unit wrap-up, and extension activities. The protocols also included areas in which evaluators could indicate which materials teachers and students used during the observed lesson. Classroom observations usually lasted approximately 40 minutes and took place during reading or writing instruction. To ensure interrater reliability, evaluators received detailed training on how to use the protocols, practiced using the protocols, and debriefed with the lead evaluator before conducting observations. Interview Protocols Evaluators conducted interviews with treatment and comparison teachers after observing their reading or writing lessons as well as with administrators and instructional support staff. To guide these interviews, evaluators developed specific protocols. The questions in the teacher protocols focused on: Evaluation of Pearson s Good Habits, Great Readers Program 6

14 1. the observed lesson 2. each teacher s class 3. each teacher s reading and writing instruction 4. each teacher s perceptions of the effectiveness the reading and writing program 5. support and professional development The questions in administrator and instructional staff protocols focused on (a) the school and reading and writing instruction at the school, (b) reading and writing instruction in the district, and (c) perceptions of the programs used by treatment and comparison teachers. Treatment and comparison teachers, administrators, and support staff participated in the spring interviews. Interviews typically lasted approximately 20 minutes. Treatment Teacher Implementation Log As part of their participation in the study, treatment teachers completed unit online implementation logs designed to measure the extent to which they used the Good Habits, Great Readers materials for their reading instruction. At the beginning of each log, teachers reported student attrition and classroom disruptions to reading or writing instruction. Teachers then responded to questions about 1. how often they had addressed specific skill areas 2. the frequency with which they had used specific program components and materials 3. whether they had used specific strategies and instructional techniques 4. if and how they had supplemented the program 5. their use of shared reading and guided reading groups 6. their use of the writing program 7. their perceptions regarding the amount of materials and pacing of instruction 8. levels of student engagement The implementation logs served several purposes in this study. First, they provided evaluators with the means to measure teachers implementation fidelity. In particular, the aggregate implementation log data yielded a way for evaluators to compute fidelity ratings specific to each teacher. These ratings enabled evaluators to compare program implementation fidelity across teachers and to the implementation guidelines set forth in the beginning of the study. The implementation logs also permitted evaluators to monitor any red flags or concerns that arose during the study period, such as when a teacher did not use required components appropriately. Evaluators could monitor logs and address these concerns immediately. Finally, implementation logs served as a feedback loop between teachers, evaluators, and program trainers, which gave teachers a process for communicating concerns and questions and provided evaluators and trainers with the means to support teachers. PROCEDURES This portion of the report describes the procedures that evaluators followed for site selection, data collection, training and implementation, and test administration and scoring. Following specific procedures ensured efficient and successful implementation of all study phases. Evaluation of Pearson s Good Habits, Great Readers Program 7

15 Site Selection In efficacy studies examining specific reading programs, it is important to ensure that participating schools accurately reflect the types of students and setting that are most likely to use the program under investigation. Therefore, for this study, evaluators considered geographic location, school size, socioeconomic status, student performance, ethnic composition, and current reading curriculum. Because there is often a learning curve for teachers who implement a program for the first time, Pearson requested that treatment-group classrooms come from schools that currently used the Good Habits, Great Readers program. Evaluators also specifically recruited treatment-group participants from schools with low student mobility. When selecting comparison-group participants, evaluators considered participants from schools with low student mobility and schools that used a comprehensive approach to instruction rather than instruction that followed a basal program. Across treatment and comparison groups, evaluators intentionally focused efforts on selecting participating schools with diverse student populations. As discussed later in the settings and participants sections, these efforts yielded a study sample that included many non-caucasian students, as well as a large number of students who were eligible to receive free or reduced-price lunch. Magnolia Consulting used many recruitment strategies to facilitate the sample selection process. In particular, Magnolia Consulting evaluators consulted with Pearson to determine what school districts currently used Good Habits, Great Readers and contacted appropriate districts from that list. In addition, evaluators pulled from its database of more than 1,000 district contacts in curriculum and instruction and identified those contacts who expressed interest in participating in a study. Evaluators also consulted with Pearson to determine which sites were priority states for recruitment and followed up with leads Pearson identified. Once Magnolia Consulting identified potential sites, evaluators contacted them about the study, provided details about what participation would involve, gauged the site s interest, and when appropriate, asked sites to complete a study application. Data Collection Timeframe Evaluators adhered to a specific timeframe for collecting data for this study. Table 1 displays this timeframe. Before the onset of the study, Magnolia Consulting conducted a study orientation, and Pearson facilitated refresher training for the Good Habits, Great Readers program and training for the writing component of the program. All teachers also received refresher DRA2 training at this time. Within the first few weeks of the start of school, all sites administered the pretest student measures (the pretest GMRT-4, the pretest DRA2, and the pretest ERAS), and then treatment teachers began implementing the Good Habits, Great Readers program, and comparison teachers implemented their regular reading programs. Treatment teacher completed end-of-unit online logs after they finished teaching each Good Habits, Great Readers unit. Treatment and comparison teachers administered the mid-year DRA2 assessment in January. In March, evaluators conducted spring site visits, which included treatment- and comparison-teacher classroom observations and interviews as well as interviews with school administrators. In May, comparison teachers completed their one-time online survey. Finally, in April, May, and June, all sites administered the posttest student measures (the posttest GMRT-4, the posttest DRA2, and the posttest ERAS). Evaluation of Pearson s Good Habits, Great Readers Program 8

16 July August September October November December January February March April May June Table 1 Timeline of Data Collection Activities TASK AND ACTIVITY Training, study orientation, study begins Administration of student measures Administration of treatment teacher implementation log (occurred after teachers implemented each unit) Administration of comparison teacher online survey Spring observations and interviews End study Test Administration and Scoring During the study orientation, evaluators provided teachers with an overview of the GMRT-4, during which they explained the general format of the test, including administration guidelines and instructions for make-up testing, accommodating students with Individualized Education Program (IEP) plans, and assessment return. After they administered the GMRT-4, teachers returned the completed assessments to site coordinators, and site coordinators then shipped the tests to Riverside Publishing scoring. After scoring the assessments, Riverside Publishing provided evaluators with electronic copies of students scores. As part of the DRA2 refresher training, Pearson provided teachers with instructions for using the DRA2, and teachers received the appropriate DRA2 materials, which included instructional manuals and other materials needed for administering and scoring the assessment. Pearson created a form for teachers to complete after each administration of the DRA2, and after completing the form with students scores, teachers entered the data online for Pearson to retrieve, or they sent the form directly to Magnolia Consulting evaluators for entry into the study database. During the study orientation, Magnolia Consulting evaluators also trained teachers in how to administer the ERAS. After each administration of the survey, teachers returned completed surveys to the site coordinator, who then sent the surveys to Magnolia Consulting for scoring and entry into the study database. Evaluation of Pearson s Good Habits, Great Readers Program 9

17 Treatment Fidelity Ensuring that treatment teachers implemented Good Habits, Great Readers with fidelity was critical to the success of this study. Therefore, Magnolia Consulting and Pearson used many implementation-monitoring techniques to ensure that treatment teachers implemented the Good Habits, Great Readers program appropriately. Specifically, teachers received Good Habits, Great Readers training, evaluators provided teachers with explicit implementation guidelines, and evaluators monitored implementation using online implementation logs, ongoing communication with sites, and site visits. Training A Pearson training consultant designed and implemented the prestudy product training for treatment teachers. 3 The trainer worked closely with evaluators to ensure that the training was consistent across study sites and that it reflected the study s research standards. Site Visits In addition to visiting sites during the study orientations, evaluators conducted spring site visits during which evaluators observed treatment and comparison teachers classrooms and conducted interviews with all teachers. These site visits helped evaluators gauge Good Habits, Great Readers program implementation and helped compare it to comparison program implementation. These spring visits provided evaluators with an abundance of qualitative data regarding program implementation in treatment and comparison classrooms. Implementation Guidelines Magnolia and Pearson collaboratively developed explicit implementation guidelines for treatment teachers to abide by throughout the study period (see Appendix D for the specific guidelines for each grade). Evaluators distributed these guidelines to treatment teachers at the study orientation, and the Pearson trainer provided clear instructions to treatment teachers for following the guidelines and implementing the program for the study. Implementation Logs As part of their participation in the study, teachers completed online end-of-unit implementation logs that generated information regarding teachers implementation of Good Habits, Great Readers. Evaluators developed these logs to align with the implementation guidelines, so they would yield a measure of each teacher s level of implementation fidelity. Evaluators closely monitored the end-ofunit logs for any indications that teachers were not adhering to the study implementation guidelines. In addition, evaluators monitored logs to determine if teachers had questions about program implementation. If concerns or questions arose about implementation, evaluators, site coordinators, and Pearson worked together to address them. Therefore, the end-of-unit implementation logs simultaneously monitored and promoted teachers implementation fidelity. 3 Comparison teachers participated in similar trainings after the study concluded. Evaluation of Pearson s Good Habits, Great Readers Program 10

18 SETTINGS Pearson Education requested that Magnolia Consulting recruit schools from different geographic regions of the country. Recruitment efforts yielded 12 participating schools across five school districts, for a total sample size of 673 students. Sites represented a wide array of districts across the nation. Two sites were located in the Mid-Atlantic region, two were in the North-Central region, and one was located in the Northeast. One district was large; two were small towns; and two were rural locations. Across the sample, several districts had a large percentage of students qualifying as low income. Additionally, the majority of districts (three out of five) were ethnically diverse. From the beginning of the study, districts did not dramatically vary on past statewide assessment performance, with four districts performing average and one performing below average on previous assessments (see Table 2). Table 2 Site Characteristics by District Site 1 Site 2 Site 3 Site 4 Site 4 Site 5 Geographic location and City description* Total student enrollment Percent qualifying as low-income Ethnic breakdown Caucasian African American Asian Hispanic Other Past performance on statewide assessments Eastern North- Central; City, Small Mid-Atlantic; Rural, Distant Mid-Atlantic; Town, Distant Northeast; City, Large Northeast; City, Large West North Central; Rural, Remote 8,824 1,458 2,361 56,765 21, % 46.77% 28.6% 49.0% 72% 41% 45.7% 37.3% 9.8% 6.8% 0.4% 56.0% 19.0% 1.0% 24.0% 0.0% 98.5% 1.1% 0.4% 0.0% 0.0% 54.0% 15.0% 8.0% 23.0% 0.0% 7.0% 61.0% 15.0% 15.0% 2.0% Average Average Average Average Below Average *City description as defined by the National Center for Educational Statistics (NCES) at % 1.0% 2.0% 11.0% 0.0% Average PARTICIPANTS This study s final analytical sample included 673 students and 44 teachers. 4 Twenty-five classrooms served as treatment classrooms, and 17 served as comparison classrooms. In 2 classrooms, two teachers co-taught, resulting in 42 classrooms. In addition, each site identified a district-level site coordinator and a school-level coordinator for each school. These coordinators were responsible for 4 Student participants included in the final sample required parental consent and at least 80% attendance over the course of the study. In addition, students needed at least one of the following sets of data: (a) two sets of DRA2 data, (b) pre and post student interest surveys, (c) pre and post GMRT data, or (d) posttest writing sample. Evaluation of Pearson s Good Habits, Great Readers Program 11

19 ensuring that treatment teachers did not share Good Habits, Great Readers materials or instructional practices with comparison teachers. Along with other responsibilities, coordinators provided evaluators with assistance in communicating any school-level issues or threats to the validity of the study. Teacher Participants Ten fourth-grade teachers and 15 fifth-grade teachers served as treatment teacher participants in the study. These teachers had used the Good Habits, Great Readers materials for at least one year before the study. As an incentive for participation, teachers received all components of the Good Habits, Great Readers Writing program free of charge (treatment teachers received materials before the start of the study, and comparison teachers received materials at the conclusion of the study). Additionally, Pearson trainers provided participant training, which normally costs an estimated $6,000, free of charge. Teachers and coordinators also received a $300 stipend for their time and efforts, 5 and Pearson provided schools with funding to cover substitute teachers during participant training. Before the commencement of the study, all teachers and coordinators completed informed consent forms. Demographics Of the 44 teachers, 31% held bachelor s degrees, 67% held master s degrees, and 2% held a 6-year certificate. Teachers had been teaching for an average of years (Range 1 38). On average, teachers taught at their current schools for 8.83 years (Range 1 38) and at their current grade level for 4.62 years (Range 0 38). Class sizes ranged from 11 to 34 students, with an average class size of 21.5 students. To ensure that the study s findings were valid, evaluators conducted chi-square analysis and t tests to determine if participant and comparison teachers differed significantly on these variables. These analyses revealed no significant differences between participant and comparison teachers for number of years teaching, t(40) = 0.76, p =.45; number of years teaching at their current grade, t(40) = 0.23, p =.82; number of years at their current school, t(39) = 0.31, p =.76; number of students per classroom, t(40) = 0.09, p =.93; and highest education degree, χ ² = 0.77, p =.68. Student Participants The study s final sample of student participants consisted of 296 fourth-grade students and 377 fifth-grade students. This section of the report presents a description of students demographic characteristics and results of analyses examining pretest equivalence between treatment and comparison groups. Demographics Table 3 presents treatment and comparison group student demographic information. Thirty-seven percent of students were in fourth grade, and 63% of students were in fifth grade. Approximately half the students (49%) were male, and half (51%) were female. Across treatment conditions, 25% 5 In one site, district regulations would not allow teachers to receive a stipend for their participation. Instead, their stipend was donated directly to the school at the end of the study. Evaluation of Pearson s Good Habits, Great Readers Program 12

20 of students were African American, 9% were Hispanic, 60% were Caucasian, and 6% were categorized as either multiracial, Asian, American Indian, or other. Thirty-six percent of the study s student participants qualified for free or reduced-priced lunch (FRL). 6 Nine percent of the study s student participants were classified as special education students, and less than 1% were classified as Section 504 students. Table 3 Student Demographics by Group Treatment Students (n = 377) Comparison Students (n = 296) Total Students (n = 673) Chi-square Results Sig. (alpha Characteristics Percent n Percent n Percent n Value = 0.05) Grade Gender 4th 37.7% % % 246 χ ² = 5th 62.3% % % Male 49.6% % % 330 χ ² = Female 50.4% % % Ethnicity African-American 11.9% % % 164 Hispanic 6.9% % % 63 Caucasian 74.5% % % 405 Other 6.6% % % 41 Free/Reduced Lunch χ ² = FRL 19.6% % % 227 χ ² = Non-FRL 80.4% % % English Proficiency LEP 1.1% 4 4.1% % 16 χ ² = Non-LEP 98.9% % % Special Education Special Ed. 10.3% % % 60 χ ² = Non-Special Ed. 89.7% % % Section 504 Sect % 3 0.7% 2 0.7% 5 χ ² = Non-Sect % % % Group Equivalence To ensure the validity of the study s findings, it is essential that the characteristics of students in treatment and comparison groups are comparable. Therefore, evaluators examined differences in demographic characteristics and attrition to determine the equivalence of students in treatment and comparison groups *.00*.02* Free or reduced-priced lunch information was not provided for 41 students in the final sample. The percentages and counts for this particular demographic do not include 6% of the sample within one site. Evaluation of Pearson s Good Habits, Great Readers Program 13

21 Student Characteristics Table 3 illustrates the results of the chi-square analyses, which indicate that in this study, treatment and comparison groups were comparable on grade level, gender, special education status, and Section 504 status. There were group differences by ethnicity, FRL status, and English proficiency. Specifically, the comparison group had a higher percentage of African-American and Hispanic students, and the treatment group had a higher percentage of Caucasian students. The comparison group also had a higher percentage of students qualifying for FRL and a higher percentage of students classifying as limited English proficient (LEP) compared to the treatment group. Comparisons of pretest ESS scores indicate that treatment and comparison groups performed differently on the GMRT Vocabulary and Comprehension tests, t(630) = 3.00, p =.003 and t(624) = 3.59, p <.001, respectively. Specifically, treatment students had significantly higher Vocabulary ESS and Comprehension ESS scores at pretest. 7 With regard to the student attitude survey, comparisons indicated that treatment and comparison groups demonstrated similar scores on recreational and academic reading attitude at pretest, t(655) = 0.82, p =.41 and t(655) = 1.89, p =.06, respectively. Attrition Evaluators conducted analyses to examine the overall sample attrition (i.e., how many participants did not complete the study for any reason), measurement attrition (i.e., how many participants did not complete all the GMRT, DRA2, or student interest survey assessments because they were absent on the day of testing), and dropout attrition (i.e., how many participants left the study) that took place throughout the study. The initial study sample included 395 Good Habits, Great Readers participants and 309 participants in comparison classrooms who had school or parent consent, for a total sample of 704 participants. The final sample for analysis consisted of 377 participants in Good Habits, Great Readers classrooms and 296 students in comparison classrooms, for 673 final study participants. The overall attrition rate was 4%, and chi-square analyses revealed no significant differences in overall attrition rates by condition, χ 2 = 0.002, p =.97. The treatment sample attrition rate was 5% (n = 18), and the comparison sample attrition rate was 4% (n = 13). The number of students dropped from the study per school ranged from one to six students. There were three reasons for overall attrition. Evaluators dropped student participants from the sample if they moved (n = 15), had incomplete data (e.g., completed only pretest and not posttest) (n = 12), or did not attend at least 80% of classes over the study (n = 4). Comparisons of pretest scores from treatment and comparison students who were dropped from the final analyses yielded no differences between treatment and comparison students. Beginning with the GMRT, there were no differences between treatment and comparison students on Vocabulary or Comprehension, t(16) = 0.47, p =.64 and t(14) = 0.49, p =.63, respectively. Additionally, on the interest surveys, there were no pretest differences for mean recreational interest scores, t(19) = 0.79, p =.44, or for mean academic interest scores, t(19) = 0.24, p = These significant differences disappeared when analyses controlled for differences in demographic variables. Evaluation of Pearson s Good Habits, Great Readers Program 14

22 PROGRAM DESCRIPTION This section of the report begins with a description of the Good Habits, Great Readers program and continues with discussions of the reading programs used by comparison-group teachers, which include Scott Foresman Reading, Spotlight on Literacy, Houghton Mifflin Reading, and homegrown materials. GOOD HABITS, GREAT READERS Good Habits, Great Readers provides comprehensive reading curriculum materials for kindergarten through fifth grade. The program supports a comprehensive approach to literacy learning through small-group instruction (guided reading) and whole-group instruction (shared reading). It is an alternative to a core reading program premised on seven research-based habits and strategies of great readers. Theoretical Foundation Good Habits, Great Readers is a comprehensive program comprised primarily of two elements of reading instruction (guided reading and shared reading) as well as literacy centers for independent and small-group work. The program provides resources for teachers who want to use small-group instruction (guided reading) and whole-group instruction (shared reading) and for teachers who prefer a comprehensive approach to literacy instruction. The Good Habits, Great Readers authors based the program on recent research identifying qualities of effective literacy teachers and classrooms as well as successful literacy strategies and routines. The program reflects the work of well-known literacy scholars, including G. S. Pinnell, I. Beck, and N. K. Duke, and it incorporates the notion that reading involves a process in which readers actively make sense of what they are reading. Good Habits, Great Readers is rooted in the theory that good readers use specific actions (good habits) while they read and that teachers can help student learn these good habits. These good habits occur before, during, and after reading. Strategies to use before reading include taking a preliminary look at the text and predicting what will happen. Strategies for use during reading include pinpointing the main idea, making inferences, and looking over the book to examine the text. Strategies for use after reading include summarizing and thinking about what happened in the text (Paris, Wasik, & Turner, 1991). The specific good habits addressed by the program are as follows: 1. Great readers see themselves as readers. 2. Great readers make sense of text. 3. Great readers use what they know. 4. Great readers understand how stories work. 5. Great readers read to learn. 6. Great readers monitor and organize ideas and information as they read. 7. Great readers think critically about texts. Evaluation of Pearson s Good Habits, Great Readers Program 15

23 Good Habits, Great Readers follows the gradual release of a responsibility model in which readers gradually gain more independence in reading. Therefore, the program seeks to provide students with the capacity to assimilate reading strategies and practices to foster independent reading (Frey, 2006). The program also seeks to build teachers capacity to be good teachers of reading comprehension by helping them use explicit teaching techniques. These techniques include providing direct explanations regarding comprehension strategies, thinking aloud to promote metacognition, and encouraging peer learning through partner talk. Program Components and Materials The Good Habits, Great Readers lessons reflect a well-defined scope and sequence with unique instructional features including (a) Focus for Instruction, (b) strategy instruction, (c) explicit modeling, (d) writing activities, (e) assessment opportunities, and (f) support for ESL/ELL students. Across grades, each unit focuses on the same strategy to provide schools with a cohesive and streamlined curriculum. At Grades 4 and 5, the Good Habits, Great Readers program includes comprehensive instructional materials to support shared reading, guided reading, and independent work as well as continuing assessment. The program teaches the following critical reading skills: (a) concepts of print, (b) phonemic awareness, (c) phonics, (d) structural analysis, (e) fluency, (f) vocabulary/word study, (g) comprehension, (h) literacy response, analysis, and appreciation, and (i) expository text. The shared reading components and materials include 25 student readers, shared text for teacher modeling, a comprehensive Teacher s Guide that provides a Five-Day Planner for each week s lessons, an audio CD of student reader selections, and a teacher resource CD. Specific components included in a shared reading lesson include lesson objectives, a minilesson, a focus lesson, opportunities for read-alouds and think-alouds, tips for teachers, embedded assessment pieces, and opportunities for differentiated instruction and for providing support to ESL/ELL students. Lesson plans also provide the means for teachers to link shared reading, guided reading, and independent reading. Guided reading components include 45 chapter books, including fiction and nonfiction and a teaching plan card for each book title. The chapter books are leveled at grade-appropriate DRA levels. The guided reading teaching plans include sections for before, during, and after reading and guide teachers through focusing students attention, working on vocabulary as well as introducing, using, and revisiting featured skills. These materials also include a reproducible graphic organizer and journal prompts. The guided reading plans also provide opportunities for further instruction, including writing activities, skill practice, and ESL/ELL support. The Good Habits, Great Readers program includes embedded assessment opportunities in all its lessons. Specific assessment tools include a comprehensive Assessment Handbook with background information on assessment, as well as user-friendly rubrics, running records, and checklists to promote ongoing student assessment. The program also includes an Assessment Card with reading prompts and checklists to guide teachers decisions regarding when students should be moved to new guided reading groups. Good Habits Great Readers provides additional materials and support. For example, the program supports ELL instruction in a number of ways. First, the scaffolding and gradual release of responsibility components incorporated into the lessons make it easy for teachers to differentiate Evaluation of Pearson s Good Habits, Great Readers Program 16

24 instruction for their ESL and ELL students. The teacher materials also include suggestions and activities for supporting these students. In addition, the program contains embedded professional development for teachers that offers best practice routines, shared reading lesson features, as well as professional articles to foster teachers professional development. The program also offers supplemental materials including the Words Their Way: Word Study in Action, the DRA2, and an online searchable database that includes all the guided reading lessons. The K 5 curriculum is divided into seven units, each of which reflects on the good habits of great readers and provides 20 days of instruction. The program covers 140 instructional days. It is recommended that teachers spend 90 minutes a day, 5 days per week, using the program components. The Good Habits, Great Readers program also has an accompanying Writing program, which includes a Teacher s Guide, a Teacher Resource CD, and a Conference Card. The Writing program is aligned with the Reading program and materials to facilitate the integration of reading and writing instruction. The Writing Teacher s Guide includes detailed writing lesson plans organized around genres and forms of writing. It provides teachers with a Pacing Chart; a Scope and Sequence chart; a description of lessons; Unit Planners; weekly extension activities; Grammar, Usage, and Mechanics Mini-lessons; and Follow-up activities and assessment resources. The program encourages teachers to spend between 30 and 45 minutes per day using these materials with their students. COMPARISON PROGRAMS Comparison teachers had access to a wide variety of materials and programs. The following section describes the core reading programs used by comparison teachers in the study. Site 2 used Scott Foresman Reading. Site 4 schools used homegrown intervention materials. Site 5 used Spotlight on Literacy and Houghton Mifflin Reading as their basal programs. The majority of comparison teachers reported heavily modifying their basal curriculums or using their own materials. This section provides a brief overview of each program that comparison teachers had access to during the study (see Appendix F for a product comparison chart). Homegrown Interventions Within Site 4, School I reported using a balanced literacy framework and leveled books. Throughout the balanced literacy process, students gradually progress from having a great deal of teacher support to being independent learners. The teacher support is removed gradually as the students acquire the strategies needed to understand the text independently. In School J, teachers used the following approaches to instruction: Teacher s College Reading and Writing Project, using Just Right Books and Calkins Art of Teaching. Sites using homegrown interventions created their own reading and writing programs using these main resources and supplemented from various sources. The Teacher s College Reading and Writing Project (Reading and Writing Project, 2010) offers student reading assessments, which evaluate concepts of print, letter/sound identification, high frequency words as well as spelling assessments, and writing assessments. The project offers Evaluation of Pearson s Good Habits, Great Readers Program 17

25 benchmarks for independent reading level progress, oral reading progress, and primary reading progress. Just Right Reading (e.g., Wutz & Wedwick, 2005) is a strategy that teaches students how to identify appropriate books for their reading level. Students use different strategies to identify when books are too easy or too hard for them. Reading "just right" books gives students the right amount of unknown words to practice applying decoding strategies, and teaching students to choose a "just right" book gets them one step closer to becoming a successful independent reader. The Art of Teaching Reading (Calkins, 2000) offers ideas about setting up classroom libraries and the value of offering students a wide array of books, reading aloud to students, and many other things that go into helping students immerse in literature. Scott Foresman Reading Scott Foresman Reading Street (PK 6) is a differentiated instruction program with ongoing, embedded assessment. Reading Street provides systematic and explicit instruction on the five core elements critical to reading success: (a) phonemic awareness, (b) phonics, (c) fluency, (d) vocabulary, and (e) text comprehension. The Reading Street program contains a fully integrated language arts strand, including writing, grammar, and spelling. Materials for instruction include three student- and teacher-leveled readers guides per grade, student and teacher practice books, grammar and writing books, a spelling Practice Book, a trade book library, and big books. The Reading Street curriculum offers a baseline group test and benchmark tests with multiple-choice questions and open-ended written responses. Each unit is meant to correspond to the skills covered in about every two chapters of the textbook. The end-of-year tests have multiple-choice questions, short-answer tasks, and open-ended written response tasks. The program also includes audio CDs for students, and teaching guide, ELL and transition handbook, posters, student readers, and 10 important sentences for ELL. Invitation to Literacy Houghton Mifflin Invitations to Literacy is a thematic integrated K 8 reading and language arts program. The program is structured around themes. It includes hands-on activities that allow students to collaborate or share information on a theme-related project with other classrooms, as well as virtual field trips to Internet sites that have content, activities, and projects related to the theme. Books for Grades 2 6 are available as six softcover individual themes or as complete hardcover anthologies. Materials for ELL include language resource bags and a Students Acquiring English Handbook. Students can practice at home with take-home booklets, Watch Me Read books, and the Phonics Home Connection. In the series, Grade 4 is entitled Imagine and includes several stories focusing on comprehension skills, such as sequencing and summarizing, and vocabulary skills, such as word and sentence structure and context clues. Assessment tools include a baseline group test and teacher's manual, cumulative evaluation record, teacher's assessment handbook, benchmark progress tests, and theme skills. Evaluation of Pearson s Good Habits, Great Readers Program 18

26 MacMillan Spotlight on Literacy Spotlight on Literacy (K 6) uses a distinct differentiated learning model. A baseline placement test is given at the beginning of the year to determine which reader a student will use. Consumable practice books are available for phonics, spelling, grammar, and comprehension. A teacher s guide is the only ELL material available in the 1997 series. Supplemental activities for small groups are a part of the program. Spotlight on Literacy is used for 32 weeks, 5 days per week, for 60 to 90 minutes per day. Houghton Mifflin Reading The Houghton Mifflin Reading (K 6) system is a thematic, differentiated learning reading program that uses Big Books, anthologies, Read-Alouds, and audio compact discs to provide step-by-step instruction in reading. According to the developer s website, Houghton Mifflin Reading was developed based on the findings of the National Reading Panel. The program is designed to be used as a fullyear curriculum program with instruction on developing oral language and comprehension, phonemic awareness, decoding skills (phonics, analogy, context, and word recognition), fluency, reading comprehension, writing, spelling, and grammar. Instruction is organized by a set of themes (10 for Grades K 1 and 6 for Grades 2 6) with selected Big Books (fiction and nonfiction literature) and other classroom activities to highlight the theme. Evaluation of Pearson s Good Habits, Great Readers Program 19

27 PROGRAM IMPLEMENTATION This section of the report begins with a description of how treatment teachers implemented the Good Habits, Great Readers program in their classrooms, which includes details about teachers material use, instructional practices, implementation fidelity, perceptions, and experiences with the program. Evaluators measured these aspects through unit implementation logs and one site visit (observations and interviews). This section of the report also provides a description of how comparison teachers implemented reading programs in their classrooms as well as their perceptions of current programs. Classroom observations, interviews, and a one-time comparison-teacher survey served as sources of data for comparison-teachers implementation. IMPLEMENTATION BY TREATMENT TEACHERS Key Question: How do teachers implement the Good Habits, Great Readers program? Treatment teachers completed online implementation logs for each unit. Within these logs, teachers provided feedback on their experiences with the Good Habits, Great Readers Reading and Writing programs. This report reflects implementation data for the entire school year. Teachers could report on seven units of the Celebration Press Reading: Good Habits, Great Readers program. Treatment teachers completed an average of six units over the study and an average of 4.8 corresponding unit logs for an overall response rate of 82%. Survey response rates varied heavily by site location. Site 1 and Site 2 averaged 6.25 and 6.16 logs, respectively, and had overall response rates of 97% or higher. In contrast, the two treatment schools in Site 4 averaged 4.38 logs (School H) and 3.25 logs (School K), with overall response rates of 77% and 54%, respectively. 8 Material Use To assess the breadth and depth of material use, teachers provided feedback on their use of program components and materials. Over the course of the school year, teachers spent an average of 3.83 days (Range 0 5) on the Good Habits, Great Readers Reading Program and an average of 3.55 days (Range 0 5) on the Writing program. In 61% of the unit logs, teachers reported that a reading or writing instructional period was missed or disrupted. Teachers attributed missed instructional periods or disruptions to a number of reasons (see Appendix X). Within the unit logs, the majority of treatment teachers (68%) reported spending minutes a week planning and preparing for the Good Habits, Great Readers lessons. An additional 14% spent less 8 Due to district research guidelines, teachers in Site 4 did not receive direct compensation or stipends for participating in the study. Therefore, because they did not receive direct compensation, teachers in Site 4 might have exhibited low motivation and interest in completing their study logs, despite numerous requests and reminders. Evaluation of Pearson s Good Habits, Great Readers Program 20

28 than 30 minutes, and 17% spent more than 90 minutes planning and preparing for the weekly lessons. In most logs (58%), teachers reported supplementing the curriculum. The majority of teachers cited supplementing the curriculum to meet student needs (n = 21) or to help students prepare for state testing (n = 14). Several other teachers supplemented the program for student interest purposes (n = 8) or because they had other valued materials (n = 6). Finally, a small handful of teachers supplemented to reinforce a concept (n = 3) or to include longer texts or novels (n = 3). Additionally, in one school at Site 4, teachers integrated reading with other content areas at least 1 day a week. The school coordinators noted, Nothing in the real world is in exclusion, and suggested that this method offers an applied look at different content areas, a bridging of science/math/reading instruction. The school created their own instructional materials for these weekly cross-applications. Teachers primarily used supplemental materials with all students (35%). However, teachers also used the materials with below-level readers (16%), advanced readers (14%), on-level readers (2%), and ELL readers (2%). Common supplemental materials included all the following: Novels or chapter books during guided or independent reading (n = 13) Higher- or lower-level books/materials (n = 10) State testing-related materials (n = 7) Newspapers (n = 6) Different guided reading or library books (n = 5) Books on special topics (e.g., fables, immigration, space) (n = 4) Different graphic organizers (n = 4) Teacher-created materials (e.g., content sheets, worksheets) (n = 3) Read-alouds (n = 3) State department of education website materials (n = 2) In addition to supplementing the curriculum, 51% of treatment teachers reported modifying specific lessons within units. Within their unit logs, teachers noted several categories of common modifications. The majority of teachers who modified the program combined or shortened lessons (n = 13) or modified all or most lessons to meet different student ability levels (n = 12). A small handful of teachers replaced components with another classroom activity (n = 2), skipped repetitive lessons (n = 2), modified lessons based on reading expertise (n = 2), used a different teacher modeling text (n = 2), or provided students with lower-level books and graphic organizers (n = 2). Instructional Practices The following section addresses use and perceptions of shared reading, guided reading, literacy centers and independent work, writing, assessment, and professional development, and online database program components. Evaluation of Pearson s Good Habits, Great Readers Program 21

29 Shared Reading During shared reading, teachers spent the most time each week on the Focus Lesson ( = 3.14) and In Action ( = 3.0) lesson components and the least time on the Celebrations ( = 1.54) program component. With regard to shared reading materials, teachers most frequently used the shared reading teacher s guide ( = 3.41) and shared text for teacher modeling ( = 3.08). Teachers spent only a few days per week using CDs, such as the Audio CD of student reader selections ( = 2.0) and Teacher Resource CD ( = 2.37). Figures F1 and F2 in Appendix F present a comprehensive description of shared reading components and weekly material use. The majority of teachers used a multitude of instructional techniques and strategies during shared reading (see Figure 1). At least 80% of teachers reported using four of six possible strategies and instructional techniques in their teaching, including applying strategies, using Think Alouds or explicit instruction, and encouraging reviewing and reflecting.. In classroom observations, evaluators confirmed that treatment teachers spent time on focus lessons during shared reading. Teachers introduced strategies, skills, and the purpose of each lesson and explained how and why good readers use skills and strategies. Most teachers linked the strategies and skills to previous instruction and used Think Alouds. The majority of classrooms were engaged during shared reading, and teachers actively interacted with students. During interviews, teachers and administrators noted that the student readers needed more fiction and variety (e.g., poetry). Teachers also reported that it was difficult to differentiate and meet all students needs with the same student reader. Additionally, several teachers indicated that they spend too much time on the same story, with one teacher noting that students lose interest because they don t want to read the same thing for four days. Asked students to apply newly learned strategies Used Think Alouds to model reading strategies Provided explicit instruction in one or more strategies Encouraged student to review and reflect on reading Provided students with prompts when they were struggling with reading 87% 87% 86% 81% 77% Used pairing strategies to increase student engagement 69% 0% 20% 40% 60% 80% 100% Figure 1. Use of instructional techniques and strategies during shared reading (n = 133). Treatment teachers also provided feedback on their use of best practice routines during shared reading. Teachers most frequently used Think-Pair-Share (78%) and Think Alouds (74%) and least frequently taught tier-two words (22%). Figure G3 in Appendix G presents a detailed depiction of Evaluation of Pearson s Good Habits, Great Readers Program 22

30 best practice routine use. During the spring interviews, teachers voiced the benefits of best practice routines, noting that these strategies often carried over to other activities, were consistent across grades, and were good at laying a foundation for study skills in future grades. One teacher noted that students, love tier-two words. They ll read these words in other areas and point them out. They also like think-pair-share because they learn from each other. It s good conversation. Some special education students experienced some difficulty with best practice routines, with one classroom teacher explaining that these students aren t there yet and cannot use these materials. Guided Reading On average, teachers used small groups to teach guided reading several days a week ( = 3.16). Teachers spent the most days each week with below-level readers ( = 2.49), followed by on-level readers ( = 2.34) and advanced readers ( = 1.99). The majority of teachers spent minutes per day of guided reading instruction in individual meetings with below-level (75%), on-level (76%), and advanced (72%) students (see Figure 2). In interviews, teachers corroborated their ratings and noted that guided reading time is heavily dependent on the skill level of the group. Table F1 in Appendix F presents additional information related to guided reading time spent on reading and time spent having conversations about reading. Less than 20 minutes minutes minutes More than 46 minutes Advanced 19% 43% 29% 9% On-Level 14% 48% 28% 10% Below-Level 13% 47% 28% 12% 0% 20% 40% 60% 80% 100% Figure 2. Length of guided reading groups (n = 131). Teachers spent the most time each week using during reading prompts ( = 2.85) and ESL/ELL support components ( = 2.69). In contrast, teachers only used journal prompts 1 day a week ( = 0.97). Teachers frequently used guided reading instructional materials, such as chapter books ( = 3.36) and the guided reading teaching plan ( = 3.20). Figure F4 in Appendix F presents a detailed depiction of guided reading component and material use. During classroom observations, evaluators observed that teachers grouped students by developmental needs, strategy proficiency, and literacy skill level. Teacher s used the guided reading cards, spent time before reading focusing student attention on what they would be reading, and introduced new vocabulary words. During reading, teachers gave students the opportunity to read independently. In approximately half the treatment classrooms, teachers prompted and checked for student understanding and followed up with support, when necessary. After reading, in Evaluation of Pearson s Good Habits, Great Readers Program 23

31 approximately half these classrooms, students had the opportunity to revisit the featured reading skill, and several teachers spent time informally assessing student learning. During spring interviews, teachers voiced their appreciation for the differentiated nature of guided reading activities, with teachers commenting on the usefulness of compare/contrast activities, cards, and organizers. Other teachers found benefits in the comprehensiveness, organization of the materials and the opportunity to have everything in one place. One teacher stated the following about her students: They are interested in reading. Every few days it s another topic with guided reading. They are making their connections, so their reading level has improved through making those connections. They practice in class and then they re applying it in their independent readers. Teachers at one school requested more advanced readers and voiced some frustration in needing to borrow Level Z books from other classrooms. Within special education classrooms, teachers offered different perceptions of guided reading materials. In one classroom, the teacher appreciated the explicit nature, vocabulary, and introduction within the lessons and noted that students express their interest by scoping out what the books are. In the other classroom, the teacher thought the books were too advanced, and they did not cater to students. Literacy Centers and Independent Work Words their Way and Quick Reads are part of the Good Habits, Great Readers program, but were not required in the study s implementation guidelines. As optional components for study implementation, only 46% of teachers reported using Words their Way sometimes or often, and 25% of teachers reported using Quick Reads sometimes or often. Teachers who did not use Words their Way or Quick Reads cited the following top three reasons for not using these program components: (a) choosing not to implement or do not have the materials (n = 58), (b) no time (n = 21), or (c) technical difficulties/issues (n = 10). Table F2 within Appendix F displays additional information on teacher use and perceptions of Words their Way and Quick Reads. Writing The majority of treatment teachers (61%) spent minutes implementing the Writing program each day. Nineteen percent of teachers spent minutes, 13% spent less than 16 minutes, and 7% spent 60 minutes or more. Within the regular writing lessons, teachers spent the greatest number of days on Apply ( = 3.42) and Lesson ( = 3.33). Teachers spent the fewest days on the Introduce lesson component ( = 2.49). As part of the Good Habits, Great Readers Writing program implementation, teachers used trait time lessons within the writing curriculum approximately six times per year. On these occasions, teachers spent the most days on ESL/ELL tips ( = 3.20) and the Grammar/Usage/Mechanics minilesson ( = 2.67) and the least days on Define ( = 1.93). Figures F5 and F6 in Appendix F present means for daily time spent on regular lesson and trait time lesson components. During classroom observations, evaluators did not directly detect the use of Good Habits, Great Readers writing materials. Instead, evaluators observed teachers using several routines common to the program. For example, teachers often demonstrated a skill or strategy, teachers and students Evaluation of Pearson s Good Habits, Great Readers Program 24

32 practiced together, and students had the opportunity to practice either independently or in pairs. After working independently or in groups, teachers often asked students to share what they had learned with the class. With regard to writing materials, teachers used the following daily instructional materials in order of daily frequency of use: Teacher s Guide ( = 3.38), Grammar, Usage and Mechanics (GUMS) book ( = 3.04), Writing Teacher Resource CD ( = 2.52) Conference Card ( = 2.49) Several teachers commented on the value of the GUMS book and noted an appreciation for their consumable nature. One teacher said that the GUMS, hits on everything we need to do with children, and another added, Kids can t get enough of it. Additionally, teachers commented on the benefits of a structured writing program and use of personal narratives and mentor texts. In contrast, some teachers said they needed more from the writing program. One asked for greater alignment to state writing assessment prompts, and others asked for additional writing products (e.g., fables, end product at end of unit, portfolio). Within special education classrooms, teachers voiced similar concerns, noting that students have trouble with the abstract ideas present in graphic organizers. One teacher noted, Writing is hard for them. They have a real mental block with it. Despite the difficulties with writing, teachers noted that they were pleased with the GUMS, and they typically send it home with students for homework. At the end of each Writing unit, teachers had the option of using several end-of-unit lesson components. Teachers used the following optional writing components: Wrap up Publishing (46%), Wrap up Informal Assessment (32%), Grammar Follow-Up Extension Activities (25%), and Digging Deeper Extension Activities (13%). The main reason teachers cited for not using end-ofunit lesson components was a lack of time or scheduling conflicts (n = 78). A small handful of teachers also mentioned a preference for different rubrics or assessment types (n = 12) or noted that the materials were not appropriate for the student ability levels in their classroom (n = 8). Additional information on what teachers liked and disliked about these components is available in Table F3 within Appendix F. Assessment Teachers had access to a variety of assessment materials in the Good Habits, Great Readers program. Teachers most frequently used the informal assessment prompts in the Shared Reading Teacher s Guide ( = 2.94), followed by the Assessment components (e.g., handbook rubrics, running records) ( = 2.77) each week. Teachers used the 4-page Assessment Card approximately 2 days a week ( = 1.93). Teachers overwhelmingly used the results of assessment to reinforce/reteach skills or strategies individually to students (74%). Only a small percentage of teachers (9%) used the online database to access lessons on specific strategies/skills as a result of assessment feedback (see Figure 3). Evaluation of Pearson s Good Habits, Great Readers Program 25

33 Worked with students individually to reinforce/reteach skills or strategies Moved student(s) up to a higher Guided Reading group level Used the Reteaching Link in the Shared Reading Teacher's Guide Moved student(s) down to a lower Guided Reading group level Accessed the online database to locate a lesson on a specific strategy/skill 9% 14% 20% 32% 74% 0% 20% 40% 60% 80% 100% Figure 3. Teacher s use of assessments to guide instruction (n = 133). Many teachers used the Assessment Handbook to drive instruction or to monitor student progress (n = 14). Other teachers noted that the assessments offered an additional guide or checklist (n = 11). One teacher explained, The checklists are a huge help. It has all of the information in one location when I am observing students. Other teachers noted that the assessments help with questioning or reviewing skills (n = 4). On a scale ranging from not at all helpful to very helpful, most teachers (59%) found the Assessment Handbook to be somewhat or very helpful. Another 20% found it not very helpful, and 21% found it not at all helpful. It is important to note that 23 teachers reported not using the handbook because it was not available or due to a use of other assessments. Fifty-two percent of teachers reported using the built-in Good Habits, Great Readers assessments to inform instruction. Teachers largely reported using these assessments as a way of determining areas where students need additional help (n = 21) or to create student groups (n = 18). A handful of teachers mentioned using results as a lesson planning tool (n = 6), and two teachers used assessments to reevaluate current teaching strategies. Teachers who did not use the built-in Good Habits, Great Readers assessments cited the following main reasons: (a) use of a different assessment model (n = 18), (b) no time (n =13), or (c) forgetting or not being aware that assessments are available (n = 9). A small handful of teachers (n = 3) reported disliking the assessments, and one teacher indicated that the program s assessments do not correlate with state testing. In spring interviews, several teachers elaborated on the lack of alignment between Good Habits, Great Readers assessments and state standardized tests. Specifically, one teacher noted that the language between assessments is different. Administrators in one treatment school voiced the same opinion and indicated that materials should support test prep spiraling. Teachers suggested creating an addition to the program that aligns embedded program assessments with state standards. In many classrooms across treatment sites, teachers reported supplementing their assessments to create greater links between program assessments and state standardized tests. Evaluation of Pearson s Good Habits, Great Readers Program 26

34 Use of Professional Development and Online Database As part of the program, professional development pieces are available for each unit, and a My Notes section is accessible, providing teachers with an opportunity to reflect on their own teaching. Additionally, teachers have access to an online database. Within the study, 42% of teachers reported using the professional development pieces, 29% used the My Notes section of the units, and 17% reported accessing the online database. Because evaluators did not require these components for study implementation, additional information on these components is available in Table F4 in Appendix F. Implementation Fidelity This section of the report compares teachers use of the program to the study s implementation guidelines. To create implementation fidelity scores for each teacher, evaluators compared teachers unit implementation log self-reports to study implementation guideline requirements for 11 variables in shared reading, guided reading, and writing. The implementation grand mean for all 25 treatment teachers was more than 100%, which indicates that overall, the teachers implemented the program appropriately. Table 4 shows that 21 (84%) of treatment teachers implemented the program with high fidelity. The remaining 4 (16%) implemented the program with moderate fidelity compared to implementation guidelines. Table 4 Number of Teachers at Three Levels of Implementation Implementation Level Overall High (80% or higher) 21 Moderate (60% 79%) 4 Low (59% or lower) Perceptions of Good Habits, Great Readers For each unit, teachers provided feedback on overall program perceptions related to the amount of material, pacing of material, and the degree to which material met student needs. Teachers also provided feedback on the overall program in their final spring interviews with evaluators. Across all seven units, teachers offered different opinions on the amount of program material. Fiftytwo percent reported there was too much to cover, 36% indicated the amount of material offered was just right and they were able to implement all or most lessons, and 11% reported that there was not enough material. With regard to pacing, most teachers reported that the overall pacing of their instruction felt reasonable (60%) or fast paced (29%) each week. A smaller percentage of teachers (11%) worked at a slow pace to meet student needs. Teachers indicated that the pace of their reading instruction somewhat (57%) or fully allowed (25%) them to address all student needs. Eighteen percent indicated they were not able to meet student needs with their pacing level. Reading. During spring interviews, teachers offered similar types of mixed feedback on the Good Habits, Great Readers program. Some teachers noted that the program skimmed the surface without Evaluation of Pearson s Good Habits, Great Readers Program 27

35 covering skills students need. One teacher elaborated that the program teaches a child, to become a better reader; skills to use to become a better reader but not necessarily skills to use as a reader. Additionally, some teachers expressed boredom with the scripted nature of the program, whereas another teacher appreciated the script, commenting, The structure is fairly uniform from K-5, making it easy to follow through the grades. Other teachers expressed an appreciation for the scripted and comprehensive lesson plans, with one teacher stating: It gives me more time in my life because I am not spending time creating lesson plans to go with books. The differentiation part is just existent within the program, because you test the kids and read with them at their level. Some teachers and administrators commented on the program emphasis on nonfiction and requested the incorporation of more fiction. One administrator noted that nonfiction materials reach more boys than girls and suggested supplementing with fiction to meet student needs. An interesting finding was that administrators across districts expressed a common overall appreciation for the balanced literacy approach embedded within the program. One administrator noted that the program is less like a traditional basal and added, no program is perfect and none will fit everyone s needs. Another administrator offered similar sentiments, stating, It is the only program I ve found that is designed around the components of balanced literacy, and another added, It is the closest thing to anything we need. Writing. During spring interviews, teachers provided overall feedback on the Good Habits, Great Readers Writing program. Teachers offered a mixed consensus on the Writing program. Some felt that the program was beneficial and provided a good plan for instruction. Teachers who liked the Writing program also felt that it addresses the traits of good writing very well. Other teachers had some concerns with the program. One teacher felt the program was not aligned to state standards, and the teacher wanted to see more writing from a prompt. Another teacher reported a need to supplement with fables, and a different teacher stated that the program was too scripted, noting that students need to dig in a little more. Finally, administrators at one school desired some type of ultimate product at the end of the year and suggested incorporating some type of portfolio writing assessment. Perceptions of Student Engagement and Interest In each unit implementation log, teachers provided feedback on the percentage of students exhibiting high, average, and low engagement levels. 9 On average, teachers indicated that the largest 9 High engagement: Students stayed on task during reading/writing instruction and enjoyed participating in the reading/writing activities. Students showed interest in the materials and seemed to love reading/writing. Students made positive comments about the materials, including the illustrations and graphics. Students often talked to each other about the materials and regularly asked questions about the reading/writing content. They requested to read/write more and wanted to read/write independently during the school day and at home. Average engagement: Students stayed on task and participated in the required reading/writing activities. They showed some interest in the materials and seemed to enjoy reading/writing. Students made some positive comments about the Good Habits, Great Readers materials. They sometimes discussed the content with each other. They did not ask to read/write more nor did they show much interest in reading/writing outside the reading/writing instructional period. Evaluation of Pearson s Good Habits, Great Readers Program 28

36 group of students exhibited high engagement, followed by average engagement. Teachers observed the largest percentage of highly-engaged students during the guided reading sessions (see Table 5). Table 5 Classroom Means of Student Engagement Levels in Shared and Guided Reading and Writing Student Engagement Level Shared Guided Writing Reading Reading High Engagement Average Engagement Low Engagement * Note: For each log, student engagement level assignments could only add to 100%, but due to the rounding of calculations over the seven units, the percentages do not add to 100. Reading. Across unit logs and interviews, teachers routinely expressed a higher level of student engagement in guided reading compared to shared reading and to writing. In guided reading, students voiced interest in and excitement over what they would be reading next. Teachers also appreciated the amount of differentiation in guided reading, noting that a small group of kids can relate to the same book and in a nonthreatening environment. Another teacher commented on the overall impact of the materials, stating, Both years [since using the program] students just loved to read. Reading is cool. The attitude grew that it is cool to read. In shared reading, teachers noted that students and teachers alike were bored with the frequency of repetitive books and lessons. One teacher stated, I almost feel bad doing these lessons because they ve done them in the third and fourth grade, so I don t know that they re getting new information. Other teachers wanted to read more of the text and expressed displeasure over reading pieces of the same book over the school year, with one teacher explaining, Don t give me a book and expect me to read it a little bit and not finish it with the kids. In life you can t do that. It needs to be more fluid. Some teachers also thought their students lost interest because they wanted to read novels instead of little books. Writing. In writing, some teachers noted that students wrote more with program support than they had in the past. In particular, teachers commented that students perceived some topics in guided reading as very motivating, which helped students generate more ideas in writing. Perceptions of Student Learning The majority of teachers (90%) reported that the materials were at least somewhat adequate in meeting the needs of on-level students. Additionally, most teachers indicated that the materials were somewhat adequate in meeting the needs of advanced (77%) and below-level readers (75%) (see Figure 4). Low engagement: Students had difficulty staying on task and participating in the required reading/writing activities. They showed very little interest in the materials and did not seem to enjoy reading/writing. Students made few or no positive comments about the Good Habits, Great Readers materials. Evaluation of Pearson s Good Habits, Great Readers Program 29

37 Very Inadequate Inadequate Somewhat Adequate Adequate Very Adequate Advanced Readers 7% 16% 28% 43% 7% On-Level Readers 5% 5% 30% 54% 6% Below-Level Readers 8% 17% 33% 37% 6% 0% 20% 40% 60% 80% 100% Figure 4. Adequacy of Good Habits, Great Readers materials in meeting needs of different groups of students (n = 129). The Good Habits, Great Readers program seeks to help students make three kinds of connections: (a) text-to-text, (b) text-to-self, and (c) text-to-world. During spring interviews, teachers offered mixed perceptions on how the program helps students make these connections. Some teachers found the program helpful for these connections, but others thought students do not have enough outside experiences to make these connections. Most teachers indicated that students do well on text-to-self and think about how it makes them feel. Teachers observed text-to-text connections in some classrooms, with some students asking to find related books on interesting materials. Teachers routinely noted that text-to-world is the most difficult, and teachers offered contrasting opinions on whether students could make these connections. With regard to special education classrooms, teachers often remarked that the shared reading and writing materials were too difficult and required extensive use of supplements. One special education classroom teacher thought the guided reading materials worked well in the classroom, and another reported that additional support was necessary for special education students. Teachers offered different perceptions of materials for lower-level readers. One teacher liked having books for the lower levels that looked like the advanced levels, which made students feel less excluded. In contrast, another teacher felt that the books are too advanced for lower-level readers and might be frustrating. For advanced readers, teachers noted that it was difficult to always find the appropriate level of materials for their highest-level students. Teachers were also divided in the perceptions of the materials for advanced students. Some teachers reported that the books challenged advanced readers because of the wide range of advanced-level books. Other teachers indicated that the materials were not challenging enough for advanced-level students, and they might have led to a lack of interest for these students. Other teachers felt the need to supplement and include more novels for their advanced readers. IMPLEMENTATION BY COMPARISON TEACHERS Comparison teachers completed a one-time log on their use and perceptions of their current core reading programs. Thirteen out of the 18 participating comparison teachers completed the survey, Evaluation of Pearson s Good Habits, Great Readers Program 30

38 for an overall response rate of 72%. Sites 2 and 5 both had overall response rates of 100%, and Site 4 had a lower response rate (64%). 10 Response rates by school were 67% for School I and 63% for School J. On average, comparison teachers reported using core reading program materials 4.54 days a week (Range 2 5) and supplementing their program 3.54 days a week (Range 0 5). Supplemental reading materials largely included supplemental reading texts (e.g., trade books, magazines, Imagine Reading, newspapers, leveled reading) (n = 9) and information from media/technology sources (e.g., websites, laptop use) (n = 2). Teachers mainly used these supplemental materials with all students (n = 10). However, a small number of teachers reported using supplements with below-level readers (n = 2), above-level readers (n = 1), on-level readers (n = 1), and ELL students (n = 1). On average, the majority of comparison teachers (77%) reported spending minutes a week planning and preparing for reading lessons. Fifteen percent reported spending more than 90 minutes a week, and 8% reported spending less than 30 minutes per week. Instructional Practices The following section addresses comparison teachers use and perceptions of shared reading, guided reading, writing, and assessment comparison program components. Shared Reading Comparison teachers spent an average of 2.92 days per week on shared reading lessons (Range 0 5). Within these daily lessons, most teachers (54%) spent minutes on shared reading instruction. Thirty-one percent spent less than 20 minutes, and 15% spend minutes. During classroom observations, evaluators noted that few comparison teachers introduced the book by building background knowledge, previewing, or making predictions. Teachers did not explain why good readers use skills and strategies and did not use think-alouds. However, the majority briefly introduced the strategies, skills, and lesson purpose. In one site, shared reading looked like small-group instruction that teachers delivered to the whole classroom, with students participating in a current novel round robin and reading aloud. At this site, the majority of teachers did not use lesson plans or have a teacher s guide. Within the one comparison special education classroom, the teacher spent whole-group time reading a story to the class and offered explicit examples of vocabulary terms within the story. The teacher provided high levels of scaffolding and assessed student comprehension of the story. Guided Reading In response to the one-time survey, comparison teachers reported spending the most days each week meeting with below-level readers ( = 2.38), followed by on-level readers ( = 1.85) and advanced readers ( = 1.62) in guided reading groups. The majority of comparison teachers spent minutes with below-level students (70%), minutes with on-level students (62%), and 10 Due to district research guidelines, teachers in Site 4 did not receive direct compensation or stipends for participating in the study. Evaluation of Pearson s Good Habits, Great Readers Program 31

39 0 30 minutes with advanced readers (76%) (see Figure 5). Additional information on time spent actually reading and time spent having conversations about reading is available in Table F5 within Appendix F. Advanced Less than 20 minutes minutes minutes More than 46 minutes 39% 39% 15% 9% On-Level 15% 62% 15% 10% Below-Level 15% 39% 31% 12% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 5. Time spent with comparison-classroom guided reading groups by reading ability level (n = 13). During classroom observations of guided reading, evaluators observed that comparison teachers differentiated instruction and introduced reading skills. However, teachers generally did not focus attention on what students would be reading or introduce new vocabulary. Students did not read independently during guided reading. In contrast to guided reading instruction within Good Habits, Great Readers classrooms, evaluators noted that guided reading time looked similar to a differentiated whole-group lesson, rather a lesson in which the teacher meets individually with small groups of students. At one comparison school, teachers indicated that they did not have the monetary resources to provide a wide range of guided reading books for their students. In the special education classroom, the teacher grouped students based on ability level and asked students to find examples of information acquired during whole-class instruction. The teacher used this time as a modified guided reading lesson and took time to offer support to each different group. Writing Comparison teachers used a variety of writing materials in their classrooms, including district writing materials at Site 2 and Workbook Plus and teacher-created materials at Site 5. In Site 4, teachers at School I used teacher-created materials and the Writer s Workshop model, and teachers at School J used the Teacher s College writing unit. On average, teachers spent 2.83 days a week using these materials (Range 1 5) and 2.44 days a week (Range 0 5) using supplemental writing materials. Supplemental writing materials included 6+1 writing traits (n = 2), writer s notebook (n = 2), and Internet resources (n = 1). The majority of teachers used writing materials with all students (n = 11), and one teacher reported using materials with ELL students (n = 1). During spring observations, evaluators only observed writing instruction at one comparison site. Within this site, students had the opportunity to practice a particular skill or strategy independently. Teachers spent the first half of instruction introducing the lesson to students and going over expectations. Students did not share with one another how they could improve their work, and teachers did not explicitly name a teaching point for the lesson. Instead, teachers asked students to write in response to a prompt and to assess their own writing. In several writing lessons at one comparison site, teachers encouraged students to reflect on their own writing, asking them to fill it Evaluation of Pearson s Good Habits, Great Readers Program 32

40 with bling, or include voice, to achieve high scores on the state writing assessment. At this same site, teachers explicitly linked writing instruction to test preparation. Teachers displayed rubrics from state assessments on transparencies and emphasized the importance of getting a high score. Comparison schools did not have formal writing curriculums and mainly followed basic models or state assessments as guides for instruction. Assessments In the one-time survey, comparison teachers reported using formal in-class assessments with students (e.g., projects/portfolios, journals, conferencing, running records) (n = 12), standardized assessments (e.g., DRA) (n = 8), and informal in-class assessments (e.g., questioning, observations) (n = 4). Teachers varied in their reports of assessment frequency. Thirty-one percent reported using assessments daily, and 31% reported conducting assessments once a week. Fifteen percent used assessments a few times each week, and 15% used them a few times each month. Finally, only 8% reported using assessments only once a month. The majority of teachers (n = 12) reported using assessment results to reinforce/reteach target skills and/or strategies with students individually. Six teachers used assessment results to move students up to a higher guided reading level, and five teachers used assessments to move students down to a lower guided reading level. During spring interviews, teachers in comparison schools confirmed the use of formal and informal assessment with their students. Beginning with formal assessments, teachers graded their students based on quizzes, end-of-unit tests, standardized testing, self-made assessments, and portfolios. Other teachers reported keeping conference notes when they work with students. With regard to informal assessment, teachers used student observations, questioning, and seat work. Across comparison classrooms and sites, there was a wide variety of approaches to assessment. Perceptions of Comparison Reading and Writing Programs Within the one-time teacher survey and during spring interviews, comparison teachers provided feedback on overall program perceptions in reading and writing related to the amount of material, pacing, and meeting student needs. Reading. With regard to reading materials, comparison teachers offered different perceptions on the amount of material provided by their current reading programs. Five teachers noted that they do not have enough materials; four noted that they have the right amount of materials; and four reported that there is too much to cover in their current reading program. Eleven teachers (85%) indicated that their reading instruction pacing is reasonable, and two teachers (15%) reported working at a slow pace. Eighty-five percent of teachers reported that their pacing somewhat (77%) or fully (8%) allows them to address student needs, and 15% said their pacing does not allow them to meet student needs. As evidenced in the descriptions of comparison reading programs, comparison teachers used a wide variety of resources. At one site, teachers used basal materials. In contrast, all other participating sites created their programs from scratch or based their programs on an overarching model. At one comparison site, teachers had access to basal materials. Teachers at this site reported an appreciation for how the current basal program lays everything out and found the leveled readers Evaluation of Pearson s Good Habits, Great Readers Program 33

41 useful. However, one teacher noted that the range of leveled readers is not wide enough, and some materials are too difficult for students. The site reading specialist also felt that their current program does not address student challenges. Other comparison sites created a curriculum entirely from scratch or designed a program around an overall model. In these sites, teachers needed to spend a lot of time creating lesson plans and pulling resources. Some teachers expressed frustration in the lack of a uniform curriculum and the need to create things from scratch or to pull resources from multiple locations. One teacher noted that for every lesson, You have to make it all yourself. With anything it is one level. So you are hitting the grade level they are supposed to be at and not the grade level they are at. Another teacher requested more guidance in the current reading program, stating: I wish we had more of a guide, like a teachers guide. Good Habits, Great Readers gives you a set of books and words you can focus on. I like that you can go back and refer to the book. It all correlates together instead of me making up the group work myself. A handful of teachers experienced difficulty in finding guided reading books for multiple levels of students and keeping an extensive library, as one teacher stated, If you re going to have balanced literacy, then your library has to be the strongest point. In contrast, some teachers did not want to give up their novels for a basal program, adding that they like that lesson decisions are up to them. One teacher expressed appreciation for teacher-driven lessons, stating, I like that I have the ability to add my own uniqueness to it. I really don t like scripted programs. I like when I can incorporate my own ideas. Other teachers appreciated that students are given more flexibility to choose interesting and engaging books, instead of being limited to a textbook. Writing. The greatest number of comparison teachers (n = 6) reported that they don t have enough writing materials; four indicated that they have too much to cover with their current writing materials, and three said they have the right amount of materials. Within the writing program, most comparison teachers reported being able to work at a reasonable pace (n = 7), with some working at a slow pace (n = 3) and others at a fast pace (n = 2). As a result of the amount of material and level of pacing, eight teachers reported that they were somewhat able to meet student needs; four said they were not able to meet student needs; and one was able to meet student needs. During spring interviews, comparison teachers offered mixed perceptions for their current writing programs or methods. Across all sites, teachers did not have a publishing writing program. The majority of teachers followed some basic writing model, but teachers indicated that their current model lacked materials, such as mentor texts to really demonstrate the different genres of writing. Another teacher requested one text that included everything, literacy and writing. Other concerns dealt with the lack of ability to differentiate for various ability levels, the lack of direction in materials, and concerns over writing topics being chosen for students by teachers. A handful of teachers appreciated the flexibility of their current writing program and being able to choose materials. One teacher elaborated, I like that it is flexible and I do what I want to do. Perceptions of Student Engagement and Interest Evaluation of Pearson s Good Habits, Great Readers Program 34

42 Similar to the engagement question in the treatment-teacher implementation logs, comparison teachers provided feedback on the percentage of students exhibiting high, average, and low engagement levels in their classrooms. 11 On average, teachers indicated that the largest group of students exhibited high engagement followed by average engagement. In contrast to the Good Habits, Great Readers classrooms, students in comparison classrooms were comparably engaged in shared reading, guided reading, and writing sessions, but also evidenced slightly lower percentages of high engagement compared to treatment classrooms (see Table 6). Table 6 Comparison Classroom Means of Student Engagement Levels in Shared and Guided Reading and Writing Student Engagement Level Shared Guided Writing Reading Reading High Engagement Average Engagement Low Engagement * Note: For each log, student engagement level assignments could only add to 100%, but due to the rounding of calculations across different teachers, the percentages do not add to 100. Reading. During the spring interviews, comparison-teacher perceptions of student engagement varied largely by school, classroom, and lesson content. Some teachers believed that their students were not engaged due to a lack of variety and limited selection of available books. To deal with these issues, some teachers allowed students to read magazines or nonfiction. In contrast, some comparison teachers thought students were engaged because they could tailor reading materials to student interest with a large library of accessible books. One teacher reported using a text-to-self connection to pull students in, adding that students, usually seem to gravitate to things they re familiar with. Another teacher noted that students were most interested in the novels they could pick rather than stories in their basal program, with one teacher elaborating, If they re interested in it, then they are more motivated to learn and performance increases as a result. Writing. With regard to writing, teachers offered mixed perceptions. Most teachers did not believe students were interested or learning a great deal in writing and requested differentiation. One teacher reported seeing some progression but wanted to see more progression. Another participant expressed concern that writing topics are limiting when chosen for students, stating, When you give them too strict of a model, it might be limiting to them. It stifles creativity. 11 High engagement: Students stayed on task during reading/writing instruction and enjoyed participating in the reading/writing activities. Students showed interest in the materials and seemed to love reading/writing. Students made positive comments about the materials, including the illustrations and graphics. Students often talked to each other about the materials and regularly asked questions about the reading/writing content. They requested to read/write more and wanted to read/write independently during the school day and at home. Average engagement: Students stayed on task and participated in the required reading/writing activities. They showed some interest in the materials and seemed to enjoy reading/writing. Students made some positive comments about the core reading materials. They sometimes discussed the content with each other. They did not ask to read/write more nor did they show much interest in reading/writing outside the reading/writing instructional period. Low engagement: Students had difficulty staying on task and participating in the required reading/writing activities. They showed very little interest in the materials and did not seem to enjoy reading/writing. Students made few or no positive comments about the core reading materials. Evaluation of Pearson s Good Habits, Great Readers Program 35

43 In contrast, a few teachers did believe students enjoyed writing. One teacher noted that after practicing with categories related to test preparation, students were interested in putting voice in their writing. Perceptions of Student Learning Within the one-time implementation log, comparison teachers responded to a set of questions related to adequacy of reading and writing materials in meeting student learning needs in reading and writing on a 1, very inadequate, to 5, very adequate, scale. The majority of comparison classrooms thought their reading materials were at least somewhat adequate in meeting the needs of on-level (85%) and advanced readers (85%). Sixty-two percent of teachers thought their materials were at least somewhat adequate in meeting the needs of below-level readers. Comparison classrooms perceived the writing materials as slightly less adequate in meeting student needs compared to reading materials. The majority of teachers (70%) found the materials inadequate to somewhat adequate in meeting below-level, on-level, and advanced student needs in writing (Table 7). Table 7 Adequacy of Comparison-Group Reading and Writing Materials in Meeting Student Needs Very Inadequate Inadequate Somewhat Adequate Adequate Very Adequate % # % # % # % # % # Reading Materials Below-Level Readers 15% 2 23% 3 23% 3 31% 4 8% 1 On-Level Readers 15% 2 46% 6 23% 3 15% 2 Advanced Readers 15% 2 62% 8 15% 2 8% 1 Writing Materials Below-Level Writers 15% 2 31% 4 39% 5 8% 1 8% 1 On-Level Writers 8% 1 31% 4 39% 5 15% 2 8% 1 Advanced Writers 8% 1 31% 4 39% 5 15% 2 8% 1 In the spring interviews, teachers provided feedback on the adequacy of their current program materials in meeting student needs. Due to the wide variety of materials and program models, teachers offered different opinions concerning the impact of materials on student learning. Beginning with advanced readers, some teachers reported that their current program challenges advanced readers if they do their part as the teacher, and many noted that these children need additional materials to be challenged. Some teachers noted that they were trying to add additional components to their teaching (e.g., learning centers, peer tutoring) to challenge these students better. With regard to lower-level readers, teachers offered mixed perceptions. Some teachers voiced disadvantages of not having enough books or their program being too difficult for lower-level readers. One teacher pointed out that students end up reading books at too difficult a level. Another teacher noted that it is often a battle to get them to read. Other teachers noted gains in student achievement and attributed these gains to tailoring book and writing options to student interests, providing daily reading worksheets, and so on. Evaluation of Pearson s Good Habits, Great Readers Program 36

44 In general, the comparison teacher did not think the program materials met the needs of special education students. Specifically, the special education teacher felt the need to reinvent the wheel and reported pulling books for students. This teacher noted that pulling folk tales and cross-cultural materials has impacted student interest and learning in reading, but noted that this might also be due to the high level of teacher enthusiasm. Treatment compared to comparison classrooms Good Habits, Great Readers provides classrooms with a balanced literacy approach to reading and writing. Compared to the teachers implementing comparison programs, teachers implementing Good Habits, Great Readers spent more time with students building background knowledge, context, and skills during shared reading lessons. Additionally, treatment teachers had access to a teacher s guide for shared reading, guided reading, and writing, but most comparison teachers did not have an explicit guide. On average, treatment teachers spent a greater amount of time differentiating instruction during guided reading and had more resources to do so than teachers in comparison classrooms had. In writing, treatment classrooms had access to a published program and offered instruction separate from state assessment requirements. The Good Habits, Great Readers program also provided these teachers with access to a wide variety of assessments. In contrast, most teachers implementing comparison programs had to create many of their own writing materials. In both treatment and comparison classrooms, teachers offered mixed perceptions on the effectiveness of their programs in different areas of reading and writing. Treatment school administrators across sites noted, No one program is perfect and none will fit everyone s needs. Treatment administrators felt that the Good Habits, Great Readers program is The only program that I ve found that is designed around the components of balanced literacy, and another added that it is the closest thing to anything we need. Evaluation of Pearson s Good Habits, Great Readers Program 37

45 STUDENT PERFORMANCE AND ATTITUDE RESULTS To address key evaluation questions regarding student learning and attitude in reading, evaluators conducted a variety of analyses, including descriptive, inferential, and multilevel modeling analyses as well as the calculation of effect sizes. This section of the report provides information about the reading performance and reading attitudes of Good Habits, Great Readers participants as well as comparisons of Good Habits, Great Readers participants and comparison-group participants. In addition, this part of the report compares differences among subgroups of study participants. LEARNING GAINS AMONG GOOD HABITS, GREAT READERS PARTICIPANTS This section of the report describes Good Habits, Great Readers participants learning gains throughout the school year. Gain scores on the GMRT-4 and DRA2 served as indicators of student learning gains. The report breaks out findings by grade and subtest, when appropriate. Descriptive Findings Key Question: Do students who participate in the Good Habits, Great Readers program demonstrate learning gains in reading during the study period? To determine the degree to which Good Habits, Great Readers program participants demonstrated learning gains over the study period, evaluators examined their pretest and posttest GMRT-4 Vocabulary, Comprehension, and Total performance as well as their pretest, midyear, and posttest DRA2 text-level scores and reading stages. Figures 6 and 7 display the mean pretest and posttest GMRT-4 Vocabulary and Comprehension grade equivalent scores for fourth- and fifth-grade Good Habits, Great Readers program participants. Figures 8 and 9 display the mean pretest, midyear, and posttest DRA2 text-level scores for fourth- and fifth-grade Good Habits, Great Readers program participants. Figure 6 shows that, on average, fourth-grade Good Habits, Great Readers participants improved their GMRT-4 Vocabulary performance by 1.7 grade levels and their GMRT-4 Comprehension performance by 0.9 grade levels. On average, fourth-grade treatment-group study participants demonstrated endof-year vocabulary and comprehension performance that exceeded their grade-level benchmarks. Figure 7 illustrates that fifth-grade Good Habits, Great Readers participants improved their GMRT-4 Vocabulary and Comprehension performance by an average of 0.6 grade levels and 0.3 grade levels, respectively, and demonstrated end-of-year vocabulary performance that exceeded their grade-level benchmark and comprehension performance that nearly met their grade-level benchmark. Evaluation of Pearson s Good Habits, Great Readers Program 38

46 8 6 4 Fourth Grade Pretest Posttest Benchmark Fifth Grade Pretest Posttest Benchmark Vocabulary Comprehension 0 Vocabulary Comprehension Figure 6. Pretest and posttest GMRT-4 Vocabulary and Comprehension grade-equivalent scores for fourth-grade Good Habits, Great Readers participants (n = 124). Figure 7. Pretest and posttest GMRT-4 Vocabulary and Comprehension grade-equivalent scores for fifth grade Good Habits, Great Readers participants (n = 181). The DRA2 text levels for grades four through eight range from Level 20 to Level 80. By the end of the school year, fourth-grade students should read at a Level 40 or above to meet the DRA2 benchmark, and fifth-grade students should read at a Level 50 or above to meet the DRA2 benchmark. Figures 8 and 9 indicate that, on average, fourth- and fifth-grade students who participated in the Good Habits, Great Readers program showed steady increases in their DRA2 textlevel scores over the study period, and their mean text-level scores were consistent with end-of-year grade-level benchmarks. 50 Fourth Grade DRA2 Text Level Benchmark 60 Fifth Grade DRA2 Text Level Benchmark Pretest Midyear Posttest Pretest Midyear Posttest Figure 8. Pretest, midyear, and posttest DRA2 text-level scores for fourth-grade Good Habits, Great Readers participants (n = 62). Figure 9. Pretest, midyear, and posttest DRA2 text-level scores for fifth-grade Good Habits, Great Readers participants (n = 162). The DRA2 classifies students into reading stages based on their text-level performance scores. The benchmark DRA2 reading stage for fourth- and fifth-grade students is the Intermediate Stage. Evaluators examined the pretest, midyear, and posttest scores of Good Habits, Great Readers program participants to determine if the percentage of students at various stages changed over the study. Figures 10 and 11 indicate that the percentage of fourth- and fifth-grade students reading at the Evaluation of Pearson s Good Habits, Great Readers Program 39

47 lower DRA2 stages (that is, Transitional and Extending) decreased over the study, and the percentage of fourth- and fifth-grade students reading at the benchmark or higher DRA2 stages (that is, Intermediate and Middle School) increased over the study. Specifically, at pretest, only 45% of fourth-grade students read at or above the benchmark Intermediate DRA2 stage, but by posttest, 81% read at or above the benchmark Intermediate DRA2 stage. For fifth grade, 76% of Good Habits, Great Readers participants read at or above the benchmark Intermediate DRA2 stage at pretest, but by posttest, 86% read at or above the benchmark Intermediate DRA2 stage. Fourth-Grade DRA2 Stages Fifth-Grade DRA2 Stages 100% 80% 60% 40% 20% 0% Transitional Extending Intermediate Middle School 2% 0% 2% 35% 18% 53% 79% 45% 65% 2% Pretest Midyear Posttest 100% 80% 60% 40% 20% 0% Transitional Extending Intermediate Middle School 6% 11% 8% 14% 9% 8% 55% 40% 70% 46% 6% 28% Pretest Midyear Posttest Figure 10. Percentage of fourth-grade treatment-group students at each DRA2 stage at pretest, midyear, and posttest (n = 62). Figure 11. Percentage of fifth-grade treatmentgroup students at each DRA2 stage at pretest, midyear, and posttest (n = 162). Multilevel Modeling Analyses In this study, students were nested in classrooms, creating a hierarchical data structure. Therefore, evaluators used multilevel modeling techniques to determine whether treatment-group students learning gains were significant. The purpose of these analyses was to determine whether gains occurred across the overall study sample of Good Habits, Great Readers program participants rather than within specific subgroups of students. Therefore, these analyses did not include covariates. Table 8 displays the results of these analyses, which revealed that, on average, students participating in Good Habits, Great Readers demonstrated gains on the GMRT-4 Vocabulary, Comprehension, and Total tests. Furthermore, the gains on the Vocabulary and Total tests were significant, corresponding to large (d = 0.56) and medium (d = 0.37) effect sizes, respectively. Gains on the Comprehension test were not significant and corresponded to a small effect size (d = 0.13).On average, Good Habits, Great Readers program participants learning gains reflect percentile gains of 21 points on the Vocabulary test, 5 points on the Comprehension test, and 14 points on the Total test. Evaluation of Pearson s Good Habits, Great Readers Program 40

48 Table 8 Mean GMRT-4 Gains for Treatment Students Standard Outcome Variable Coefficient Error t- value Approx. df p- value Effect Size Percentile Gain GMRT-4 Vocabulary * GMRT-4 Comprehension GMRT-4 Total * * Significant at the.05 level Subgroup Performance Based on Student Demographic Characteristics Key Question: Is participation in the Good Habits, Great Readers program associated with differential reading gains for different subgroups of students? Evaluators conducted additional multilevel modeling analyses to determine if various student, teacher, and setting characteristics were associated with differential gains among students participating in the Good Habits, Great Readers program. Specifically, evaluators included the following covariates to examine gains within subgroups: gender, ethnicity, FRL status, LEP status, and special education status. Evaluators calculated effect sizes to provide insight regarding the practical importance of these findings. Table G1 in Appendix G, displays the results of these analyses. Because the subgroup analyses divide the sample, readers should use caution when interpreting the results of these analyses. It is important to note that these subgroup analyses did not examine whether each subgroup of students made significant gains. Instead, these analyses examined whether or not there were differential gains across subgroups. In other words, they examined whether or not one subgroup demonstrated gains that were significantly larger or smaller than another group. Therefore, a negative coefficient does not mean that a specific subgroup did not make significant learning gains, but rather, it indicates that a specific subgroup gained less than the subgroup to which it was compared. 12 The subgroup analyses revealed no significant differences by gender, ethnicity, or FRL status on GMRT-4 Vocabulary, Comprehension, or Total test gains. These findings indicate that males and females demonstrated similar gains, Caucasian and non-caucasian students demonstrated similar gains, and students eligible and not eligible for FRL demonstrated similar gains on all GMRT-4 subtests. These analyses, however, did reveal significant differences by LEP status and special education status. In particular, students with LEP gained significantly more than students who were proficient in English on the GMRT-4 Vocabulary (d = 0.60), Comprehension (d = 0.78), and Total (d = 0.78) tests, and special education students demonstrated significantly greater gains on the GMRT-4 Vocabulary (d = 0.22) and Total (d = 0.29) tests than non-special education students. 12 For example, Table G1 reveals a negative coefficient for grade. This does not mean that fifth-grade students did not demonstrate learning gains. Instead, it means that fifth-grade students learning gains, on average, were less than those demonstrated by fourth-grade students. Evaluation of Pearson s Good Habits, Great Readers Program 41

49 Subgroup Performance Based on Grade Evaluators also used multilevel modeling analyses to determine whether Good Habits, Great Readers participants grade level in school (that is, whether they were in fourth or fifth grade) was associated with differential learning gains. The findings revealed that compared to fourth-grade Good Habits, Great Readers participants, fifth-grade participants gained significantly less on the GMRT-4 Vocabulary (d = 0.71) and Total (d = 0.65) tests. Fifth-grade participants also gained less on the GMRT Comprehension test (d = 0.47), and although the difference was not significant, the effect size was notable. As with other subgroup analyses, readers should use caution when interpreting this finding because the analysis divided the sample. Subgroup Performance Based on Initial Reading Stage In this study, students read at various DRA2 stages (that is, Transitional, Extending, Intermediate, or Middle School) at the onset of the study. Figures 12 and 13 display students pretest and posttest mean GMRT-4 Vocabulary and Comprehension scores by initial DRA2 reading stage. These figures suggest similar gains across student subgroups based on initial DRA2 reading stage. Furthermore, a one-way between-groups analysis of variance 13 revealed that all groups of students demonstrated comparable gains on the GMRT-4 Vocabulary [F(3, 203) = 0.18, p =.91], Comprehension F(3, 201) = 0.96, p =.41], and Total [F(3, 201) = 1.00, p =.40] tests. This finding suggests that the Good Habits, Great Readers program might be equally effective for students regardless of whether they begin the school year reading at Transitional, Extending, Intermediate, or Middle School DRA2 reading stages Transitional Intermediate Pretest Extending Middle School Posttest Transitional Intermediate Pretest Extending Middle School Posttest Figure 12. Mean GMRT-4 Vocabulary pretest and posttest scores by initial DRA2 reading stage. Figure 13. Mean GMRT-4 Comprehension pretest and posttest scores by initial DRA2 reading stage. 13 Evaluators did not use multilevel modeling due to the small number of teachers per group for these subgroups. Because the analysis of variance did not account for the nested data structure, readers should use caution when interpreting the results. Evaluation of Pearson s Good Habits, Great Readers Program 42

50 To determine whether each group of students (that is, students who read at each DRA2 reading stage at pretest) demonstrated significant gains from pretest to posttest, evaluators calculated paired samples t tests. 14 Table 9 reveals that all groups of students demonstrated significant learning gains on the GMRT-4 Vocabulary and Total tests, and all students, except those reading at a transitional stage at pretest, demonstrated significant learning gains on the GMRT-4 Comprehension test. It is important to note that in all cases, the effect sizes were medium to large, and even though students who initially read at a transitional stage did not demonstrate significant gains, the gains they demonstrated corresponded to a large effect size. Table 9 Paired Samples t tests for Treatment Students Performance within Initial DRA Reading Stage Initial Reading Stage N Pre-Post Mean Difference SD t-value df p-value ES (Cohen's d) Percentile Gain GMRT-4 Vocabulary Transitional * Extending * Intermediate * Middle School * GMRT-4 Comprehension Transitional Extending * Intermediate * Middle School * GMRT-4 Total Transitional * Extending * Intermediate * Middle School * * Significant at the.05 level 14 Using multilevel modeling was not appropriate for this analysis because of the small number of teachers per subgroup. Because the t tests did not account for the nested data structure, readers should interpret results cautiously. Evaluation of Pearson s Good Habits, Great Readers Program 43

51 Subgroup Performance Based on District To examine whether there were significant differences by district in Good Habits, Great Readers participants gains on the GMRT-4 Vocabulary, Comprehension, or Total tests, evaluators conducted a one-way between-groups analysis of variance with post-hoc tests. 15 Results indicated a statistically significant difference for Vocabulary gains for the three districts [F(2, 305) = 3.14, p =.05]. Post-hoc comparisons revealed that, on average, students in District 1 (M = 10.86, SD = 23.31) gained significantly less than students in District 4 (M = 19.08, SD = 26.78). On average, students in District 3 (M = 13.52, SD = 18.34) demonstrated comparable gains to those in Districts 1 and 4. Subgroup Performance Based on Teacher Characteristics Evaluators used multilevel modeling to determine whether the type of college degree or years of teaching experience affected student gains on the GMRT-4 Vocabulary, Reading, or Total tests. These analyses revealed no significant differences by teacher degree or by years of teaching experience on the Vocabulary, Reading, or Total test. Evaluators also used multilevel modeling to examine whether the degree to which teachers implemented the Good Habits, Great Readers program with fidelity contributed to differential student gains on the GMRT-4. Table 10 displays these findings, which revealed that although higher implementation fidelity was associated with higher gains, the relationship was not statistically significant. Table 10 Effect of Implementation Fidelity on Treatment Student Mean GMRT-4 Gains Outcome Variable Coefficient Standard Error t- value Approx. df p- value GMRT-4 Vocabulary GMRT-4 Comprehension GMRT-4 Total Taken together, the results from the subgroup analyses by teacher characteristics suggest that the Good Habits, Great Readers program might work equally well for students, regardless of the teacher characteristics examined in these analyses. As with all subgroup analyses, readers should interpret these findings cautiously because the analyses divide the sample. CHANGES IN READING ATTITUDE AMONG GOOD HABITS, GREAT READERS PROGRAM PARTICIPANTS This section of the report describes Good Habits, Great Readers program participants changes in reading attitudes over the study period. Evaluators assessed student attitudes toward reading with 15 Using multilevel modeling was inappropriate for this analysis because of the small number of teachers per district for several districts. Because this analysis divided the sample and did not account for the nested data structure, readers should interpret results cautiously. Evaluation of Pearson s Good Habits, Great Readers Program 44

52 the ERAS, which includes measures of attitude toward recreational reading and academic reading. The ERAS also includes a score for total attitude toward reading. Key Question: Do students who participate in the Good Habits, Great Readers program demonstrate a change in their attitude toward reading over the course of the study? How does their attitude relate to their reading achievement? Descriptive Findings Figure 14 displays Good Habit, Great Readers participants mean pretest and posttest scores for Recreational Reading Attitude, Academic Reading Attitude, and Total Reading Attitude. The possible range for these scores is 1 to 4, in which a 1 indicated less a less positive attitude in reading, and a 4 indicated a more positive attitude toward reading. The pretest and posttest means suggest that although, on average, these students reading attitude scores slightly decreased over the study, they remained on the higher end of the spectrum, suggesting that students participating in the Good Habits, Great Readers program remained at least somewhat positive in their attitudes toward reading throughout the study. When interpreting the findings for reading attitude, it is important to note that the survey that assessed reading attitude used a simple, 4-point rating scale, which provided a straightforward scale for fourth- and fifth-grade students, but also limited the range of possible responses. Furthermore, teachers administered the posttest attitude survey very close to the end of the school year when students were likely anticipating summer break. Therefore, it is important to interpret these findings with caution. 4 Recreational Academic Total Pretest Posttest Figure 14. Mean pretest/posttest Reading Attitude scores for students participating in Good Habits, Great Readers (n = 287). Evaluators also examined students average responses to each of the specific ERAS questions that assessed attitude toward recreational and academic reading. Figure 15 displays the findings for questions assessing attitude toward recreational reading, and Figure 16 displays the findings for questions assessing attitude toward academic reading. On average, Good Habits, Great Readers participants responses generally reflected a moderate interest in all assessed recreational reading situations at pretest and posttest, with the exception of two situations: (a) reading instead of playing and (b) reading during summer vacation. Likewise, on average, their responses to academic reading Evaluation of Pearson s Good Habits, Great Readers Program 45

53 situations generally reflected moderately positive attitudes, but they seemed to have less positive attitudes toward reading tests and reading workbook pages and worksheets than other situations. Attitude Toward Recreational Reading Pretest Posttest About reading different kinds of books About going to a bookstore About reading instead of playing About reading during summer vacation About starting a new book About spending free time reading a book About getting a book for a present About reading for run at home When you read a book in school during free time When you read a book on a rainy Saturday 29% 27% 80% 79% 88% 83% 45% 39% 90% 85% 61% 53% 67% 62% 63% 57% 72% 60% 66% 66% 0% 20% 40% 60% 80% 100% Figure 15. Percentage of Good Habits, Great Readers program participants who indicated that they felt extremely positive about various situations involving recreational reading. Evaluation of Pearson s Good Habits, Great Readers Program 46

54 Attitude Toward Academic Reading Pretest Posttest About taking a reading test About using a dictionary When you read out loud in class About stories you read in reading class When it's time for reading in class About learning from a book About Reading your school books About reading in school About reading workbook pages and worksheets When a teacher asks you questions about what you read 43% 41% 42% 40% 57% 53% 54% 56% 76% 69% 74% 66% 78% 76% 61% 54% 84% 75% 62% 54% 0% 20% 40% 60% 80% 100% Figure 16. Percentage of Good Habits, Great Readers program participants who indicated that they felt extremely positive about various situations involving academic reading. Multilevel Modeling Analyses Evaluators used multilevel models to determine whether the pretest to posttest changes in reading attitude among Good Habits, Great Readers program participants were significant. Because these analyses sought to determine whether changes were significant for the general sample of Good Habits, Great Readers program participants rather than for specific subgroups of students, these analyses did not include covariates. Table 11 displays these results, which indicate that, on average, Good Habits, Great Readers program participants demonstrated significant decreases in Recreational Reading Attitude over the course of the study. This decrease corresponded to a small effect size (d = 0.26), equivalent to an average percentile loss of 10 points. Changes in Academic Reading Attiutde were not significant among Good Habits, Great Readers participants and corresponded to a small effect size (d = 0.10) and an average percentile loss of 4 points. As mentioned previously, readers should interpret findings for reading interest with caution due to potential measurement limitations. Table 11 Mean ERAS Gains for Treatment Students Standard Outcome Variable Coefficient Error t- value Approx. df p- value Effect Size Percentile Loss Recreational Reading * Academic Reading Total Reading * Significant at the.05 level Evaluation of Pearson s Good Habits, Great Readers Program 47

55 Association between Reading Attitude and Learning in Reading Among Treatment Students To examine whether there was an association between changes in Academic Reading Attitude and changes in reading performance among Good Habits, Great Readers participants, evaluators used multilevel modeling with Academic Reading Interest as the independent variable and reading performance gain as the dependent variable. 16 Table 12 displays the results of these analyses, which indicate that among Good Habits, Great Readers program participants, changes in Academic Reading Attitude were not significantly associated with gains on the GMRT-4 Vocabulary, Comprehension, or Total tests. Therefore, these findings suggest that students participating in the Good Habits, Great Readers program demonstrate comparable learning gains in reading, regardless of whether their attitude toward Academic Reading changed over the study period. Table 12 Association Between Reading Attitude and Learning in Reading Outcome Variable Coefficient Standard Error t- value Approx. df p- value GMRT-4 Vocabulary GMRT-4 Comprehension GMRT-4 Total STUDENT LEARNING COMPARISONS FOR READING PERFORMANCE The following section of the report presents the results from comparisons of Good Habits, Great Readers participants and comparison-group participants gains on the GMRT-4 and DRA2 assessments. The primary purpose of these analyses was to determine whether Good Habit, Great Readers participants showed comparable learning gains to students participating in other high-quality reading programs. Key Question: How does the reading performance of students who participate in the Good Habits, Great Readers program compare to that of students participating in other reading programs? Descriptive Findings Evaluators examined descriptive statistics to compare treatment-group and comparison-group students pretest to posttest performance on the GMRT-4 Vocabulary and Comprehension tests. Figures 16 Evaluators ran one model with GMRT-4 Vocabulary gain as the dependent variable, one model with GMRT-4 Comprehension as the dependent variable, and one model with GMRT-4 Total as the dependent variable. Evaluation of Pearson s Good Habits, Great Readers Program 48

56 17 through 20 compare the mean raw scores on the GMRT-4 Vocabulary and Comprehension tests. Examination of these means suggest that treatment-group and comparison-group students demonstrated very similar pretest to posttest slopes, suggesting that their changes from pretest to posttest were comparable. 17 Fourth-Grade Comprehension Fourth-Grade Vocabulary Treatment Comparison Treatment Comparison Pretest Posttest 10 Pretest Posttest Figure 17. Fourth-grade pre/post GMRT-4 Vocabulary raw scores for treatment-group (n = 124) and comparison-group (n = 93) students. Figure 18. Fourth-grade pre/post GMRT-4 Comprehension raw scores for treatment-group (n = 124) and comparison-group (n = 93) students. Fifth-Grade Comprehension Fifth-Grade Vocabulary 40 Treatment Comparison 40 Treatment Comparison Pretest Posttest 10 Pretest Posttest Figure 19. Fifth-grade pre/post GMRT-4 Vocabulary raw scores for treatment-group (n = 181) and comparison-group (n = 168) students. Figure 20. Fifth-grade pre/post GMRT-4 Comprehension raw scores for treatment-group (n = 181) and comparison-group (n = 168) students. Evaluators also examined pretest, midyear, and posttest mean DRA2 text-reading levels among fourth- and fifth-grade treatment-group and comparison-group students. Figures 21 and 22 display these means and suggest that, on average, in both fourth and fifth grade, Good Habits, Great Readers 17 It is important to note that these descriptive analyses do not control for pretest differences between treatment and comparison groups. Therefore, readers should use caution when interpreting these data. Evaluation of Pearson s Good Habits, Great Readers Program 49

57 program participants and comparison-group participants demonstrated comparable changes in their DRA2 text-level scores from pretest to posttest Fourth-Grade DRA2 Text-Level Treatment Comparison Fifth-Grade DRA2 Text-Level Treatment Comparison Pretest Midyear Posttest Pretest Midyear Posttest Figure 21. Fourth-grade DRA2 text-level scores for treatment-group (n = 62) and comparisongroup (n = 99) students. Figure 22. Fifth-grade DRA2 text-level scores for treatment-group (n = 162) and comparison-group (n = 187) students. In addition to comparing treatment- and comparison-group students GMRT-4 scores and DRA2 text levels, evaluators also compared their shifts in DRA2 reading stages throughout the study period. Figures 23 and 24 indicate that, on average, among fourth-grade students, there was a lower percentage of treatment-group students (45%) than there were comparison-group students (62%) reading at or above benchmark DRA2 stages (that is, the Intermediate and Middle School stages). However, by posttest, there was a slightly higher percentage of treatment-group students (81%) than there were comparison-group students (78%) reading at or above benchmark DRA2 stages. Therefore, among fourth-grade study participants, there was a larger increase in the percentage of students reading at or above benchmark DRA2 stages from pretest to posttest among Good Habits, Great Readers participants than among comparison-group participants. Figures 25 and 26 indicate that among fifth-grade study participants, the percentage of Good Habits, Great Readers participants reading at or above benchmark DRA2 stages increased from 76% to 86%, and in comparison groups, this percentage increased from 86% to 94%. Therefore, although a greater percentage of fifth-grade comparison-group, compared to treatment-group, students were reading at or above benchmark DRA2 stages at the end of the study, the pretest to posttest percentage increase was higher for Good Habits, Great Readers participants These descriptive analyses do not control for preexisting group differences. Therefore, readers should use caution when interpreting these data. 19 It is important to note that these descriptive analyses do not control for pretest differences between treatment and comparison groups. Therefore, readers should use caution when interpreting these data. Evaluation of Pearson s Good Habits, Great Readers Program 50

58 Fourth-Grade Treatment DRA2 Stages Fourth-Grade Comparison DRA2 Stages Transitional Extending Transitional Extending 100% 80% 60% 40% 20% 0% Intermediate 45% 65% Middle School 2% 0% 2% 53% 35% 18% Pretest Midyear Posttest 79% 2% 100% 80% 60% 40% 20% 0% Intermediate Middle School 11% 6% 5% 20% 17% 27% 62% 74% Pretest Midyear Posttest 69% 9% Figure 23. Percentage of fourth-grade treatmentgroup students at each DRA2 stage at pretest, midyear, and posttest. Figure 24. Percentage of fourth-grade comparison-group students at each DRA2 stage at pretest, midyear, and posttest. 100% 80% 60% 40% 20% 0% Fifth-Grade Treatment DRA2 Stage Transitional Extending Intermediate Middle School 6% 11% 8% 8% 14% 9% 40% 70% 6% 55% 28% Pretest Midyear Posttest 46% 100% 80% 60% 40% 20% 0% Fifth-Grade Comparison DRA2 Stage Transitional Extending Intermediate Middle School 1% 2% 2% 11% 6% 1% 53% 75% 63% 44% 12% 30% Pretest Midyear Posttest Figure 25. Percentage of fifth-grade treatmentgroup students at each DRA2 text level at pretest, midyear, and posttest. Figure 26. Percentage of fifth-grade comparisongroup students at each DRA2 text level at pretest, midyear, and posttest. Evaluation of Pearson s Good Habits, Great Readers Program 51

59 Multilevel Modeling Analyses Due to the hierarchical structure of these data, evaluators used multilevel modeling analyses to determine whether any differences in learning gains between Good Habits, Great Readers program participants and comparison-group program participants were significant. Specifically, these analyses compared learning gains among treatment- and comparison-group students on the GMRT-4 Vocabulary, Comprehension, and Total tests. 20 Table 12 displays the results of these analyses, which reveal that Good Habits, Great Readers participants gained less than comparison-group participants on the GMRT-4 Vocabulary, Comprehension, and Total tests, but no differences were significant, and they corresponded to small effect sizes (d = 0.20, d = 0.21, and d = 0.24, respectively). These findings suggest that students who participated in Good Habits, Great Readers demonstrated gains on the GMRT-4 comparable to those of students who participated in comparison programs during the study period. Therefore, whether a student participated in Good Habits, Great Readers or in a comparison program did not have a significant impact on learning gains measured by the GMRT-4. Table 12 Effect of Condition on Mean GMRT-4 Gains Standard Outcome Variable Coefficient Error t- value Approx. df p- value Effect Size Percentile Difference GMRT-4 Vocabulary GMRT-4 Comprehension GMRT-4 Total * Significant at the.05 level Effect of Condition on Various Subgroups of Students Evaluators examined whether any program effects on learning gains between treatment and comparison students differed across various subgroups of students, including subgroups defined by gender, ethnicity, FRL status, LEP status, and special education status. Each model controlled for preexisting demographic differences between treatment and comparison groups and included the interaction term of interest as well as the variables contributing to the interaction. 21 Because these analyses divide the sample, readers should interpret findings with caution. Table G4, G5, and G6, in Appendix G, display these results, which reveal significant study condition by gender, study condition by LEP status, and study condition by special education status interactions for the GMRT-4 Vocabulary and Total tests as well as significant study condition by LEP status and study condition by special education status interactions for the GMRT-4 Comprehension test.. It is important to note that these subgroup analyses did not examine whether each subgroup of students made significant gains within each condition. Instead, these analyses examined whether there were differential gains for subgroups by study condition. For example, the condition by gender analyses examined whether participating in the treatment versus comparison group had a significant impact 20 These analytic models included covariates that controlled for treatment- and comparison-group differences in ethnicity, FRL status, and LEP status. To avoid issues of multicollinearity and to facilitate interpretations, they did not include other covariates. 21 These analyses included covariates to control for treatment- and comparison-group differences in ethnicity, FRL status, and LEP status. To avoid issues of multicollinearity and to facilitate interpretations, these models did not include other covariates. Evaluation of Pearson s Good Habits, Great Readers Program 52

60 on the degree to which there were differential gains for males compared to females. Figures are provided to facilitate interpretations of significant interactions. Figures 27 and 28 display the means for the variables contributing to the significant study condition by gender interaction on the GMRT-4 Vocabulary and Total tests and suggest that in the treatment group, males and females demonstrated similar learning gains, but in the comparison group, females demonstrated higher gains than males. These findings suggest that the comparison programs were more effective for females than the Good Habits, Great Readers program. 25 Male Female Male Female Treatment Comparison 0 Treatment Comparison Figure 27. Study condition by gender interaction on GMRT-4 Vocabulary test gains. Figure 28. Study condition by gender interaction on GMRT-4 Total test gains. Figures 29, 30, and 31 display the means for the variables contributing to the significant study condition by LEP status interaction on the GMRT-4 Vocabulary, Comprehension, and Total tests. These significant interactions suggest that treatment-group students with LEP gained more on the GMRT-4 than did comparison-group students with LEP. These findings suggest that the Good Habits, Great Readers program was more effective for students with LEP than were comparison programs. Therefore, the Good Habits, Great Readers program shows promise in potentially helping to close the reading achievement gaps between students with LEP and those proficient in English LEP Treatment Non-LEP Comparison LEP Treatment Non-LEP Comparison Figure 29. Study condition by LEP interaction on GMRT-4 Vocabulary test gains. Figure 30. Study condition by LEP interaction on GMRT-4 Comprehension test gains. Evaluation of Pearson s Good Habits, Great Readers Program 53

61 LEP Non-LEP Treatment Comparison Figure 31. Study condition by LEP interaction on GMRT-4 Total test gains. Figures 32, 33, and 34 highlight the means for the variables contributing to the significant study condition by special education status interaction on the GMRT-4 Vocabulary, Comprehension, and Total tests. These findings suggest that in the treatment group, special education and non-special education students demonstrated comparable gains, but in the comparison group, non-special education students gained more on the GMRT-4 than did special education students. These findings suggest that the Good Habits, Great Readers program shows promise in potentially helping to close the reading achievement gaps between special education students and non-special education students. SPED Non-SPED SPED Non-SPED Treatment Comparison -3-8 Treatment Comparison Figure 32. Study condition by special education status interaction on GMRT-4 Vocabulary gains. Figure 33. Study condition by special education status interaction on GMRT-4 Comprehension gains. Evaluation of Pearson s Good Habits, Great Readers Program 54

62 SPED Non-SPED Treatment 1.01 Comparison Figure 34. Study condition by special education status interaction on GMRT-4 Total test gains. COMPARISONS IN STUDENT READING ATTITUDES Evaluators used multilevel modeling analyses to examine how changes in reading attitudes of students participating in Good Habits, Great Readers compared to those of students participating in comparison programs. Table 13 displays the results of these analyses and indicates that Good Habits, Great Readers program participants gained less than comparison program participants did in Recreational Reading Attitude, Academic Reading Attitude, and Total Reading Attitude. These differences were not significant and corresponded to small but notable effect sizes (d = 0.19 for Recreational Reading Attitude, d = 0.26 for Academic Reading Attitude, and d = 0.26 for Total Reading Attitude). These findings suggest that, on average, whether a student participated in Good Habits, Great Readers versus comparison programs did not significantly affect changes in reading attitude over the study period. 22 Table 13 Effect of Condition on Mean ERAS Gains Standard Outcome Variable Coefficient Error t- value Approx. df p- value Effect Size Percentile Difference Recreational Reading Academic Reading Total Reading Readers should use caution when interpreting the findings for reading interest due to potential measurement limitations. Evaluation of Pearson s Good Habits, Great Readers Program 55

63 STUDENT END-OF-YEAR WRITING COMPARISONS As indicated previously, treatment-group and comparison-group students completed a writing assessment at the end of the study period. Therefore, evaluators used multilevel modeling to examine whether there were end-of-year differences in writing performance among Good Habits, Great Readers students who used the writing component of the program and comparison-group students. 23 The outcome variables for these analyses included standardized scores for Writing Ideas, Writing Organization, Writing Voice, Writing Word Choice, Writing Sentences, Writing Conventions, and a Writing Total score. 24 Table 14 displays the results of these findings and indicates that although there were no significant differences between Good Habits, Great Readers participants and comparison-program participants, there were notable effect sizes for some writing subtests. In particular, the effect sizes for Writing Ideas and Writing Word Choice were above 0.20 and favored treatment-group students (d = 0.26 and d = 0.23, respectively), and the effect size for Writing Convention was notable and favored comparison-group students (d = 0.26). These findings suggest that, on average, although Good Habits, Great Readers students displayed greater focus and elaborations (Ideas) and clearer sequences and moods (Word Choice) than comparison students, and although comparison students displayed fewer errors (Conventions), treatment and comparison groups generally demonstrated comparable end-of-year writing performance. Table 14 Effect of Condition on Mean End-of-Year Writing Performance Outcome Variable Coefficient Standard Error t- value Approx. df p- value Effect Size Percentile Difference Writing Ideas Writing Organization Writing Voice Writing Word Choice Writing Sentences Writing Convention Writing Total These analyses controlled for pretest group difference in ethnicity, FRL, and LEP. 24 Appendix B provides a summary of the traits represented by each score. Evaluation of Pearson s Good Habits, Great Readers Program 56

64 SUMMARY & DISCUSSION The purpose of this study was to examine the efficacy of Pearson s Good Habits, Great Readers curriculum materials for fourth- and fifth-grade students. Good Habits, Great Readers is a comprehensive program that facilitates balanced literacy instruction through resources to support shared reading (whole-group instruction), guided reading (small-group instruction), and independent reading. The Good Habits, Great Readers program also includes a Writing program that aligns closely to the reading program components. Overall, treatment teachers implemented the Good Habits, Great Readers program with fidelity. Specifically, 21 teachers implemented the program with high fidelity, and 4 implemented the program with moderate fidelity. Participating classrooms fully implemented the shared reading and guided reading lessons, in addition to the Writing program and assessment components. Many teachers reported supplementing the Good Habits, Great Readers materials to help them meet students needs and to help them prepare for state testing. There was no significant relationship between implementation and student learning, which suggests that the Good Habits, Great Readers program might be equally effective, regardless of whether teachers implement it with high or moderate fidelity. Teachers shared feedback about the Good Habits, Great Readers program through implementation logs, interviews, and ongoing correspondence with evaluators. Overall, teachers views of the program were mixed, with many teachers appreciating the structure and organization of the materials, and other teachers wanting more out of a reading and writing program. Teachers who used Good Habits, Great Readers over the study period found two program components to be most effective in improving student s reading skills: the guided reading component and the best practice routines. Regarding the guided reading component, teachers appreciated the organization and comprehensiveness of guided reading materials, which allowed them to differentiate instruction to meet the needs of diverse students. During classroom observations, evaluators observed this differentiated instruction and noted that it occurred more often in Good Habits, Great Readers classrooms than in comparison classrooms. In regards to best practice routines, teachers valued the consistency of routines across grades, and some teachers noted that the strategies might lay the foundation for key study skills in high school. This suggests that some of the strategies Good Habits, Great Readers participants learn in the elementary years might have the potential to benefit their learning and achievement when they engage in other learning tasks later in their schooling. Although treatment teachers perceptions of the program were mixed, school administrators indicated that the overall program met their needs. Administrators also acknowledged that no single program could meet the needs of all students, demonstrating their understanding of the reality that it is unlikely that any given program would ever meet all pedagogical needs of diverse educators and learners. Many comparison teachers who were using a variety of other programs also found their programs lacking in some ways, such as not providing a uniform curriculum or enough materials. Students in Good Habits, Great Readers classrooms generally displayed at least average levels of engagement in the Shared Reading, Guided Reading, and Writing program components, with higher levels of engagement evident in the Guided Reading lessons than in Shared Reading or Writing lessons. Therefore, the differentiated aspect of guided reading that allows teachers to tailor lessons Evaluation of Pearson s Good Habits, Great Readers Program 57

65 to reading levels might also facilitate student engagement in the lessons. Some teachers expressed concerns about the lack of novels in the shared reading portion of the program and thought that might negatively impact student engagement. In general, teachers found the Good Habits, Great Readers program at least adequate in meeting the needs of all their students. The study findings indicated that, on average, students participating in the Good Habits, Great Readers program demonstrated significant gains on the Vocabulary and Total tests over the study period. These gains corresponded to large and medium effect sizes and average percentile gains of 21 points and 14 points. Furthermore, at the end of the study period, fourth-grade and fifth-grade students who participated in the program were reading at or above their benchmark DRA2 levels, on average. Results suggested that participating in Good Habits, Great Readers was not associated with differential learning gains by gender, ethnicity, or eligibility to receive free or reduced-price lunch, but that it was associated with differential gains by English proficiency and special education status. Specifically, participating in Good Habits, Great Readers was associated with higher gains on the GMRT-4 Vocabulary, Comprehension, and Total tests among students with LEP (compared to those who were proficient in English), as well as higher gains on the GMRT-4 Vocabulary and Total tests for special education students (compared to non-special education students). Because students often begin the school year at various reading levels, this study also examined whether students at various reading levels gained equally over the study period. The findings revealed that participating in Good Habits, Great Readers was not associated with differential learning gains by beginning-of-year DRA2 reading stages. This finding suggests that the program might be equally effective for students regardless of their initial reading level. On average, Good Habits, Great Readers program participants demonstrated significant decreases in Recreational Reading Attitude over the study. This difference corresponded to a small effect size and is difficult to interpret due to possible measurement limitations (such as a limited response range and a potential testing effect associated with the end of the school year). Treatment students changes in Academic Reading Attitude were not significant. Examination of the relationship between reading attitude and achievement found no association between reading attitude and reading gains, suggesting that the program might be equally effective for students regardless of their attitude toward reading. Good Habits, Great Readers participants and comparison-group participants demonstrated comparable gains on the GMRT-4 Vocabulary, Comprehension, and Total tests. This finding suggests that students participating in Good Habits, Great Readers experience reading learning gains comparable to students who participate in other high-quality reading programs. Analyses examining subgroup effects by study condition suggest that comparison programs might have been more effective than Good Habits, Great Readers for females. Subgroup effects also emerged regarding English proficiency status and special education status. Specifically, the Good Habits, Great Readers program might have been more effective than comparison programs for students with LEP, as well as for special education students. These findings suggest that the Good Habits, Great Readers program shows promise in potentially helping to close the reading achievement gaps between students with LEP and students proficient in English as well as the achievement gaps between special education student and non-special education students. Compared to students participating in comparison programs, there was a greater increase in the percentage of students who moved from reading below benchmark DRA2 reading stages at the beginning of the study to reading at or above benchmark DRA2 reading stages at the end of the study period. Therefore, the Good Habits, Great Readers program might be more effective Evaluation of Pearson s Good Habits, Great Readers Program 58

66 than comparison programs at helping students read at or above their benchmark DRA2 reading stages. Analyses comparing reading attitude gains of Good Habits, Great Readers program participants to those of comparison-program participants suggested that the type of program in which a student participated had no significant effect on their reading attitude over the study period. The study findings regarding treatment and comparison students writing achievement were mixed. Although there was no significant difference between Good Habits, Great Readers participants and comparison-program participants writing performance, there were notable effect sizes for some of the writing subtests. In particular, the effect sizes for Writing Ideas and Writing Word Choice favored treatment-group students (d = 0.26 and d = 0.23, respectively), and the effect size for Writing Convention favored comparison-group students (d = 0.26). These findings suggest that although treatment and comparison students demonstrated comparable end-of-year writing performance, on some measures of writing (Writing Ideas and Writing Word Choice), Good Habits, Great Readers students performed better than comparison-program participants, and on one measure of writing (Writing Convention), comparison students outperformed treatment students. In conclusion, this study s findings revealed that teachers liked many aspects of the Good Habits, Great Readers program, but they also found the program lacking in some ways. This finding is likely typical for most reading programs, as it would be very difficult for a single program to meet the needs of all educators and students. Administrators acknowledged this reality and indicated that, in general, Good Habits, Great Readers met their needs. Regarding student learning, on average, fourthand fifth-grade students participating in the Good Habits, Great Readers program over the school year demonstrated learning gains in reading, some of which were significant. Furthermore, their learning gains were comparable to those of students participating in other high-quality reading programs. Future research examining the cumulative learning effects of Good Habits, Great Readers over a longer duration would facilitate a deeper understanding of the program s effectiveness. Evaluation of Pearson s Good Habits, Great Readers Program 59

67 REFERENCES Calkins, L. M. (2000). The art of teaching reading. New York, NY: Longman. Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N. A., & Chambers, B. (2005). The National Randomized Field Trial of Success for All: Second-Year outcomes. American Educational Research Journal. Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (3 rd ed., pp ). New York: Macmillan Publishing Co. Frey, N. (2006). Celebration Press: Good Habits, Great Readers The research guiding an innovative program for instruction in reading comprehension. Retrieved from Lyons, C. A., & Pinnell, G. S. (2001). Systems for change in literacy education. Portsmouth, NH: Heineman. McKenna, M. C, & Kear, D. J. (1990). Measuring attitude toward reading: A new tool for teachers. The Reading Teacher, 43, Neuman, S. B., & Dickinson, D. K. (Eds.) (2001). Handbook of early literacy research. New York, NY: Guilford Publications. National Institute of Child Health and Human Development (NICHD). (2001). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: National Institutes of Health. Paris, S. G., Wasik, B. A., & Turner, J. C. (1991). The development of strategic readers. In R. Barr, M. L. Kamil, P. Mosenthal, and P. D. Pearson (Eds.), Handbook of reading research (Vol. II, pp ). Mahwah, NJ: Lawrence A. Earlbaum. Pearson Education, Inc. (2007). Celebration Press Reading: Good Habits, Great Readers Program overview. Parsippany, NJ: Pearson Education. Raudenbush, S., Spybrook, J., Liu, X. F., & Congdon, R. (2005). Optimal design for longitudinal and multilevel research. [computer software and documentation for computer software]. New York, NY: William T. Grant Foundation. Reading and Writing Project. (2010). The Teacher s College reading and writing project: Columbia University. Retrieved from Shannon, L. C., Wilkerson, S. B., & Herman, T. L. (2007). A final report for the evaluation of Pearson Education s Celebration Press Reading: Good Habits, Great Readers Program. Louisa, VA: Magnolia Consulting, LLC. Vaughn, S. & Linan-Thompson, S. (2004). Research-based methods of reading instruction, Grades K 3. Alexandria, VA: Association for Supervision and Curriculum Development. Evaluation of Pearson s Good Habits, Great Readers Program 60

68 Wutz, J. A., & Wedwick, L. (2005). Scaffolding book selection for independent reading. The Reading Teacher, 59(1), Evaluation of Pearson s Good Habits, Great Readers Program 61

69 APPENDIX A Data Analysis Plan Summary Data Analysis Plan Summary Evaluation Question 1a. What are the characteristics of students who participated in the study? 2. What are the characteristics of teachers who participated in the study? 3. Are there demographic differences between treatment- and comparison-group students? 4. Do students who participate in the Good Habits, Great Readers program demonstrate significant learning gains in reading during the study period? Analysis Participant Characteristics Descriptive statistics for grade, gender, ethnicity, free/reduced lunch, migrant status, disabled status, special education status, and limited English proficiency (LEP) Descriptive statistics for number of years teaching, number of years teaching at current grade, highest degree, and level of implementation Chi-square analyses with treatment/comparison group as the independent variable and gender, ethnicity, free and reduced lunch, migrant, disabled, special education, and LEP as the dependent variables Treatment Student Performance Evaluators will use HLM 6 software to examine GMRT gains among treatment students from pretest to posttest. The outcome variables will be pretest to posttest gains in GMRT Vocabulary and Comprehension test scores. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + r ij Level 2 equation: β 0j = γ 00 + u 0j Evaluators will conduct descriptive analyses of DRA2 data to examine treatment students growth in text level over the study and will examine pretest to posttest shifts in DRA2 reading stages over the study. 5. Do students who participate in the Good Habits, Great Readers program demonstrate changes in reading attitudes over the study? How do reading attitudes relate to reading achievement? Evaluators will use HLM 6 software to examine attitude changes among treatment students from pretest to posttest. The outcome variables will be pretest to posttest changes in reading attitudes. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + r ij Level 2 equation: β 0j = γ 00 + u 0j Evaluators will include students attitude changes as covariates in a subsequent analysis to determine the impact of this gain on their reading gains

70 Evaluation Question 6. Does participating in the Good Habits, Great Readers program have differential effects for various subgroups of students and for students in different settings? 7. What is the magnitude of the gains experienced by treatment students? 8. How does the reading performance of children participating in the Good Habits, Great Readers program compare to that of children participating in other reading programs? 9. How do changes in reading attitudes among students who participate in the Analysis Evaluators will use HLM 6 software to examine GMRT gains from pretest to posttest (while including covariates to examine subgroup effects) among treatment students. The outcome variables will be pretest to posttest gains in GMRT Vocabulary and Comprehension test scores. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + β 1jX 1ij + β 2jX 2ij + + β QjX Qij + r ij Level 2 equation: β qj = γ q0 + γ q1w 1j + γ q2w 2j + + γ qsqw sqj +u qj β qj= γ q0 Level 1 predictor variables Gender Ethnicity Special education status Limited English proficiency Initial reading ability Level 2 predictor variables Implementation fidelity District Calculate effect sizes for student performance results on the GMRT using the formula for Cohen s and the formula from Raudenbush et al. (2005), as appropriate. t c Raudenbush et al. (2005) X X Cohen s d s t s 2 Treatment and Comparison Group Comparisons c Evaluators will use HLM 6 software to examine the effect of condition on GMRT gains from pretest to posttest. The outcome variables will be pretest to posttest gains in GMRT Vocabulary and Comprehension test scores. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + r ij Level 2 equation: β 0j = γ 00 + γ 01W j + u 0j Level 1 predictor variables β 001 Level 2 predictor variables Condition Evaluators will conduct descriptive analyses of DRA2 data to examine treatment versus comparison students growth in text level over the study and will examine treatment- and comparison-group students pretest to posttest shifts in DRA2 reading stages over the study. Evaluators will use HLM 6 software to examine the effect of condition on reading attitude changes from pretest to posttest. The outcome variables will be pretest to posttest changes in reading attitudes. The

71 Evaluation Question Good Habits, Great Readers program compare to those of students participating in other programs? equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + r ij Level 2 equation: β 0j = γ 00 + γ 01W j + u 0j Analysis Level 1 predictor variables Level 2 predictor variables Condition 10. Are the effects of participating in the Good Habits, Great Readers program versus comparison programs different for various subgroups of students and across settings? 11. How does the writing performance of students who participate in the Good Habits, Great Readers program compare to that of students who participate in other reading programs at the conclusion of the study? 12. What is the magnitude of performance differences between treatment and comparison Evaluators will use HLM 6 software to examine the effect of condition on GMRT gains from pretest to posttest (while including covariates to examine subgroup effects). The outcome variables will be pretest to posttest gains in GMRT Vocabulary and Comprehension test scores. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + β 1jX 1ij + β 2jX 2ij + + β QjX Qij + r ij Level 2 equation: β qj = γ q0 + γ q1w 1j + γ q2w 2j + + γ qsqw sqj +u qj β qj= γ q0 Level 1 predictor variables Gender Ethnicity Special education status Limited English proficiency Initial reading ability Level 2 predictor variables Condition District School Evaluators will use HLM 6 software to examine the effect of condition on posttest writing scores. The outcome variables will be writing assessment scores. The equations for these analyses are as follows: Level 1 equation: Y ij = β 0j + r ij Level 2 equation: β 0j = γ 00 + γ 01W j + u 0j Level 1 predictor variables Level 2 predictor variables Condition Calculate effect sizes for student performance results using the formula for Cohen s and the formula from Raudenbush et al. (2005), as appropriate

72 groups? Evaluation Question 13. What is the breadth and depth of teachers use of the Good Habits, Great Readers program materials and components? 14. Do teachers implement the program with fidelity? 15. What are teachers perceptions and experiences with the materials and components? Cohen s d X s t t X c s 2 Treatment Teacher Program Implementation c Analysis Raudenbush et al. (2005) - Descriptive statistics for implementation log data, calculated percentage of implementation compared to benchmark - Content analysis and analytic induction of participant interviews and classroom observations β

73 APPENDIX B Assessment Reliability Information GMRT Reliability The GMRT, Fourth Edition was the standardized norm-referenced diagnostic assessment used in this study. Evaluators administered Form S in the fall and Form T in the spring. Fourth-grade students completed Level 4, and fifth-grade students completed Level 5. Level 4 contains 93 items and takes 55 minutes to complete, and Level 5 contains 93 items and takes 55 minutes to complete. Riverside Publishing scoring services provided separate scores for Vocabulary, Comprehension, and Total. The Kuder-Richardson Formula 20 (K-R 20) reliability coefficients for the GMRT Level 4 raw scores and Level 5 raw scores are presented in Table B1. The K-R 20s for the Level 4 and Level 5 Vocabulary test, Comprehension test and Total test are all above.91, indicating high reliability. Reliability Information for the GMRT Level Form Test # of items K-R 20 4 S Vocabulary Comprehension Total T Vocabulary Comprehension Total S Vocabulary Comprehension Total T Vocabulary Comprehension Total DRA2 Reliability The DRA2 is the individually administered assessment used in this study that measures student performance in the following areas: (a) reading engagement, (b) oral reading fluency, and (c) comprehension. This assessment is fully integrated into the Good Habits, Great Readers program. Reliability information from 2006 is based on a sample of 1,676 students in Grades K 8. Cronbach s alpha on the subscales of oral fluency and comprehension ranged from.72 to.79 at Levels Elementary Reading Attitude Survey (McKenna & Kear, 1990) The Elementary Reading Attitude Survey (McKenna & Kear, 1990) was designed to assess student emotion toward reading, specifically academic and recreational reading. Using a sample of more than 18,138 students in 1989, the authors calculated reliability statistics. In fourth and fifth grades, Cronbach s alpha on the subscales (recreational and academic) and full scale ranged from.81 to.89. Writing Assessment Evaluation of Pearson s Good Habits, Great Readers Program

74 An evaluator with classroom experience in reading and writing graded all writing assessments. This evaluator was blind to student condition and used the Good Habits, Great Readers short-story rubric (within the rubrics for Anchor papers) for Grades 4 and 5. This rubric provides a score of 1 4 for six writing traits, including 1. Ideas assesses degree to which story has focus and contains elaborations 2. Organization assesses degree to which the story is organized and has clear time points 3. Voice assesses degree to which the story has a believable character or narrator 4. Word choice assesses the degree to which the story has a sequence and clear mood 5. Sentence fluency assesses the degree to which the story has different and well-constructed sentences 6. Conventions assesses the frequency of errors within the story The evaluator read each narrative twice, once before scoring and once after assigning scores to the six different traits. The following are the narrative prompts for Grades 4 and 5: Grade 4 Narrative (Arkansas State Test) Suppose you are on a flying carpet that takes you anywhere you choose. Think about where you would go and what you would do. Now, write a story about your ride on a flying carpet. Give enough detail so that the person reading your story will understand what happened. Grade 5 Narrative (Maine State Test) While hiking on a trail with your friends, you discover a hidden cave. Write a story about what happens next. Evaluation of Pearson s Good Habits, Great Readers Program

75 APPENDIX C Implementation Guidelines Celebration Press Good Habits Great Readers Study Implementation Guidelines Grade 4 Teachers should implement the following instructional routines during their daily Literacy Block: Shared Reading (Whole Class): minutes per day Guided Reading (Small Group and Individual Work): minutes per group 60 minutes a day Writing Instruction: o Preferred minutes a day outside the 90-minute reading block, using the Celebration Press Good Habits Great Readers Writing program. o Minimum 30 minutes 3 days a week outside the 90-minute reading block using the Celebration Press Good Habits Great Readers Writing program. Required Daily Routines for Good Habits Great Readers Day 1 Shared Reading Introduce the Teacher Modeling Text and/or Student Reader Mini Lesson Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Focus Lesson 1 Apply the Strategy Day 2 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Evaluation of Pearson s Good Habits, Great Readers Program

76 Shared Reading Focus Lesson 2 Apply the Strategy Day 3 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Focus Lesson 3 Apply the Strategy Mini Lesson Day 4 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Pause and Reflect or Celebrations Write About It Day 5 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Guided Reading Groups: Once a week, teachers should be meeting with Guided Reading groups for the following durations: Below-Level Readers: At least three 20-minute periods a week On-Level Readers: At least three 20-minute periods a week Above-Level Readers: At least three 20-minute periods a week Evaluation of Pearson s Good Habits, Great Readers Program

77 Additional Required Materials and Components for Good Habits, Great Readers DRA2 Online Database Assessment Handbook Optional Materials and Components for Good Habits, Great Readers Words Their Way Quick Reads Options for Further Instruction Section at the end of the Guided Reading lessons Graphic Organizers at the end of the Guided Reading Lessons Teacher Resource CD-Rom Required Routines for Good Habits Great Readers Writing The Good Habits Great Readers Writing program includes four lessons per week. Some lessons might require more than 1 day to complete. Every week, implement the following: Lesson 1 Prewriting, Trait Time, Drafting, or Revising (depending on the unit) Lesson 2 Prewriting, Trait Time, Drafting, or Revising (depending on the unit) Lesson 3 Prewriting, Drafting, Revising, or Editing (depending on the unit) Lesson 4 Prewriting, Drafting, Revising, Editing, or Publishing (depending on the unit) Additional Required Materials for Good Habits, Great Readers Writing Teacher Resource CD Writing Conference Prompts Within the Lessons Evaluation of Pearson s Good Habits, Great Readers Program

78 Optional Materials for Good Habits, Great Readers Writing Grammar, Usage, Mechanics, and Spelling Workbooks Extension Activities at the end of Each Unit Writing Conference Card Celebration Press Good Habits, Great Readers Study Implementation Guidelines Grade 5 Teachers should implement the following instructional routines during their daily Literacy Block: Shared Reading (Whole Class): minutes per day Guided Reading (Small Group and Individual Work): minutes per group 60 minutes a day Writing Instruction: o Preferred minutes a day outside the 90-minute reading block, using the Celebration Press Good Habits, Great Readers Writing program. o Minimum 30 minutes 3 days a week outside the 90-minute reading block, using the Celebration Press Good Habits, Great Readers Writing program. Required Daily Routines for Good Habits, Great Readers Day 1 Shared Reading Introduce the Teacher Modeling Text and/or Student Reader Mini Lesson Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Focus Lesson 1 Apply the Strategy Day 2 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Evaluation of Pearson s Good Habits, Great Readers Program

79 Day 3 Shared Reading Focus Lesson 2 Apply the Strategy Mini Lesson (occasionally on Day 3, instead of Day 4) Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Focus Lesson 3 Apply the Strategy Mini Lesson Day 4 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Shared Reading Pause and Reflect or Celebrations Write About It Day 5 Guided Reading Groups Meet with at least two Guided Reading groups, while other students participate in independent reading/writing Engage in Before Reading, During Reading, and After Reading activities Guided Reading Groups: Once a week, teachers should be meeting with Guided Reading groups for the following durations: Below-Level Readers: At least three 20-minute periods a week On-Level Readers: At least three 20-minute periods a week Above-Level Readers: At least three 20-minute periods a week Evaluation of Pearson s Good Habits, Great Readers Program

80 Additional Required Materials and Components for Good Habits, Great Readers DRA2 Online Database Assessment Handbook Optional Materials and Components for Good Habits, Great Readers Words Their Way Quick Reads Options for Further Instruction Section at the end of the Guided Reading lessons Graphic Organizers at the end of the Guided Reading Lessons Teacher Resource CD-Rom Required Routines for Good Habits, Great Readers Writing The Good Habits, Great Readers Writing program includes four lessons per week. Some lessons might require more than 1 day to complete. Every week, implement the following: Lesson 1 Prewriting, Trait Time, Drafting, or Revising (depending on the unit) Lesson 2 Prewriting, Trait Time, Drafting, or Revising (depending on the unit) Lesson 3 Prewriting, Drafting, Revising, or Editing (depending on the unit) Lesson 4 Prewriting, Drafting, Revising, Editing, or Publishing (depending on the unit) Additional Required Materials for Good Habits, Great Readers Writing Teacher Resource CD Evaluation of Pearson s Good Habits, Great Readers Program

81 Writing Conference Prompts Within the Lessons Optional Materials for Good Habits, Great Readers Writing Grammar, Usage, Mechanics, and Spelling Workbooks Extension Activities at the end of Each Unit Writing Conference Card Evaluation of Pearson s Good Habits, Great Readers Program

82 APPENDIX D School-Level Characteristics Site 1 Site 2 Site 3 School A Grades K 5 School B Grades K 5 School C Grades K 5 District Totals School D Grades K 5 District Totals School E Grades K 5 School F Grades K 5 School G Grades K 5 District Totals School Assignment Treatment Treatment Treatment Control Treatment Treatment Treatment Fourth Grade Classrooms Number of Students Fifth Grade Classrooms Number of Students School Totals Classrooms Number of Students Gender Among Participants Female 55.3% 45.8% 43.5% 50.0% 56.1% 56.1% 38.5% 55.1% 45.5% 49.6% Male 44.7% 54.2% 56.5% 50.0% 43.9% 43.9% 61.5% 44.9% 54.5% 50.4% Ethnicity Among Participants African American 44.7% 33.3% 26.1% 37.2% 22.0% 22.0% 0.0% 0.0% 4.5% 0.9% Hispanic 0.0% 0.0% 13.0% 3.2% 9.8% 9.8% 0.0% 2.9% 0.0% 1.7% Caucasian 46.8% 62.5% 43.5% 50.0% 63.4% 63.4% 100.0% 92.8% 95.5% 94.9% Asian/Pacific Islander 8.5% 4.2% 17.4% 9.6% 0.0% 0.0% 0.0% 1.4% 0.0% 0.9% Other 0.0% 0.0% 0.0% 0.0% 4.9% 4.9% 0.0% 2.9% 0.0% 1.7% Limited English Proficiency Among Participants LEP 0.0% 0.0% 8.7% 2.1% 9.8% 9.8% 0.0% 0.0% 0.0% 0.0% Non-LEP 100.0% 100.0% 91.3% 97.9% 90.2% 90.2% 100.0% 100.0% 100.0% 100.0% Special Education Among Participants Special Education 6.4% 0.0% 4.3% 4.3% 7.3% 7.3% 15.4% 23.2% 0.0% 17.1% Non-Special Education 93.6% 100.0% 95.7% 95.7% 92.7% 92.7% 84.6% 76.8% 100% 82.9% Free/Reduced Price Lunch Among Participants Free/Reduced Lunch 19.1% 33.3% 13.0% 21.3% 53.0% % 0.0% 20.3% 0.0% 12.0% Non-Free/Reduced Lunch 80.9% 66.9% 87.0% 78.7% 47.0% 47.0% 100.0% 79.7% 100.0% 88.0% Section 504 Among Participants Section % 0.0% 4.3% 3.2% 4.9% 4.9% 0.0% 0.0% 0.0% 0.0% Non-Section % 100.0% 95.7% 96.8% 95.1% 95.1% 100.0% 100.0% 100.0% 100.0% 25 Information only available at the school level Evaluation of Pearson s Good Habits, Great Readers Program

83 Site 4 Site 5 School H Grades K 5 School I Grades K 5 School J Grades K 5 School K Grades K 5 District Totals School L Grades K 5 District Totals Study Totals School Assignment Treatment Control Control Treatment Control Fourth Grade Classrooms Number of Students Fifth Grade Classrooms Number of Students School Totals Classrooms Number of Students Gender Among Participants Female 49.6% 49.1% 53.6% 54.7% 51.7% 48.9% 48.9% 51.0% Male 50.4% 50.9% 46.4% 45.3% 48.3% 51.1% 51.1% 49.0% Ethnicity Among Participants African American 8.0% 8.8% 94.5% 0.0% 35.4% 1.1% 1.1% 24.4% Hispanic 18.6% 26.3% 2.7% 0.0% 11.7% 17.0% 17.0% 9.4% Caucasian 63.7% 50.9% 0.0% 96.2% 45.6% 78.4% 78.4% 60.2% Asian/Pacific Islander 8.8% 10.5% 0.0% 3.8% 5.4% 3.4% 3.4% 4.6% Other 0.9% 3.5% 2.7% 0.0% 1.8% 0.0% 0.0% 1.5% Limited English Proficiency Among Participants LEP 0.9% 7.0% 1.8% 1.9% 2.4% 2.3% 2.3% 2.4% Non-LEP 99.1% 93.0% 98.2% 98.1% 97.6% 97.7% 97.7% 97.6% Special Education Among Participants Special Education 9.7% 7.0% 5.5% 7.5% 7.5% 9.1% 9.1% 8.9% Non-Special Education 90.3% 93.0% 94.5% 92.5% 92.5% 90.9% 90.9% 91.9% Free/Reduced Price Lunch Among Participants Free/Reduced Lunch 31.0% 0.0% 100.0% 9.4% 45.0% 48.9% 48.9% 35.9% Non-Free/Reduced Lunch 69.0% 100.0% 0.0% 90.6% 55.0% 51.1% 51.1% 64.1% Section 504 Among Participants Section % 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.7% Non-Section % 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 99.3% Evaluation of Pearson s Good Habits, Great Readers Program

84 Component APPENDIX E Content Analysis of Comparison Curriculum Table August 2010 Homegrown Interventions MacMillan Spotlight on Literacy 1997 Scott Foresman Reading Street 2007 Houghton Mifflin Reading Grade Levels Multiple K 6 PK 6 K 6 Focus Balanced literacy, differentiated learning, independent reading strategies. Very strong differentiated learning program Differentiated instruction with ongoing, embedded assessment; 3-Tier model Thematic, differentiated learning Program Components Reading comprehension Independent learning skills Language skills Comprehension Vocabulary Guided reading Guided writing Guided reading Phonics Language arts Vocabulary Fluency Gr. 1, 2, and 3: vocabulary, build background, fluency Gr. 1 and 2: share literature, phonemic awareness, phonics, highfrequency words, comprehension skills and strategies, guided comprehension, speaking and listening Gr. 3: build concept, skills, and strategies in context; prereading strategies; reader response Oral language and comprehension Phonemic awareness Decoding skills Fluency Reading comprehension Writing Spelling Grammar Materials (Note: Materials listed do not necessarily include all materials available from each publisher, especially Various reading and writing materials Leveled readers, theme big books, theme support lessons. Phonics and decoding book starter. Literacy support book Leveled Readers Teacher s Guide and Student editions Graphic organizer book Gr. 2 6, centers survival kit; Readers Theater Themes: Big Books, anthologies, Read Alouds, and audio compact discs to provide step-by-step instruction in reading. One anthology for each grade. Evaluation of Pearson s Good Habits, Great Readers Program

85 items available from all three, such as the Teacher s Edition and Student Edition, transparencies, blackline masters, graphic organizers.) starter. Vocabulary/comprehens ion book. Practice: Spelling practice book, Grammar practice book, Phonics practice book, Literacy launch practice book, Phonics activity book. Read Alouds, Plays, and Choral Readings. ELL: teacher s guide Assessment: Diagnostic/placement evaluations, early literacy assessment, performance assessment, progress assessment, selection and unit assessments. Anthology Practice: Practice Book S.E. and T.E., Grammar and Writing Book and Practice Book, Spelling Practice Book Trade Book Library Big Books; ELL/Struggling readers: Teaching guide, ELL and Transition Handbook, posters, student readers, Ten Important Sentences, Audio CDs: audio text Gr. K 6; build background Gr. 3 6 Assessment: Baseline Group Test, Unit Tests, End-of-Year Benchmark Tests, Fresh Reads for Differentiated Test Practice, Selection Tests Software CD-ROMs: Exams Online Access Packs all S.E., Teacher Resources (all T.E., leveled readers, planner), Leveled Readers, Success Tracker (assessment) Grades 2 6 have six themes each. Four main leveled readers. Classroom management kit. Teaching resource kit, Learner profile, classroom intervention kit, lesson planner CD-ROM. Practice: Student practice book, handbook practice, technology support, leveled readers practice. ELL: Handbook for English Language Learners. Assessment: Phonics/Decoding screening test, baseline group test, Lexia Quick Phonics CD- Rom, Leveled reading Passages assessment kit. Instructional Time Use to replace or enhance comprehension components of other programs 32 weeks, 5 days/week, minutes per day 150 instructional days; 5 days a week, minutes per day Grade 4: 6 units Grade 5: 6 units Evaluation of Pearson s Good Habits, Great Readers Program

86 Component Houghton Mifflin Invitations to Literacy Pearson Education s Celebration Press Reading: Good Habits, Great Readers 2007 Grade Levels K 8 K 5 Focus Program Components Materials (Note: Materials listed do not necessarily include all materials available from each publisher, especially items available from all three, such as the Teacher s Edition and Student Edition, transparencies, black-line masters, graphic organizers.) Thematic integrated reading and language arts program Reading Comprehension Vocabulary Social development skills Phonics Technological skills Grades 2 6 books are available as six softcover individual themes or as complete hardcover anthologies. ELL: Language Resource Bags, Students Acquiring English Handbook Practice: Take-Home Booklets, Watch me read books, Take-Home Booklets, Phonics Home Connection Assessment: Baseline Group Test and Teacher's Manual, Cumulative Evaluation Record, Teacher's Assessment Balanced literacy approach using shared reading and guided reading Guided reading Shared reading Phonics Phonemic awareness Vocabulary Fluency Comprehension Skills and strategies Shared Reading materials: Big books and trade books, wall-sized poetry posters, audio CDs, Teacher s Guide Guided Reading materials: leveled little books (K 5), teaching plan folders for each book title Practice: Teacher s Guide provides suggestions and activities for student practice, including Literacy Centers (Audio Centers, Fluency Practice, Vocabulary Builder, Writing Center, Word Study, Response Center) ELL: Teacher s Guide provides suggestions and activities to support ELL Audio CDs: audio selections from student readers (4 5), audio CD of Evaluation of Pearson s Good Habits, Great Readers Program

87 Handbook, Benchmark Progress Tests and Teacher's Annotated Edition, Theme Skills Tests and Teacher's Annotated Edition big books (K 3) Assessment: Teacher handbook, Assessment card suggesting when to move student to next DRA2 level Other: Online searchable database; Words Their Way: Word Study in Action; QuickReads Instructional Time Grade 4: 6 units Grade 5: 6 units Seven units, each with 20 days of instruction; 90 minutes per day, 5 days per week Evaluation of Pearson s Good Habits, Great Readers Program

88 APPENDIX F Supporting Text, Figures, and Tables for Program Implementation The following appendix provides additional information left out of the body of the report, for space and coherence purposes. The following sections address missed/disrupted periods, in addition to the use of materials in shared reading, guided reading, Words their Way and Quick Reads, writing, supplemental program components, and guided reading time in control classrooms. Missed/Disrupted Instructional Periods In 61% of the unit implementation logs, treatment teachers reported that they missed an instructional period or that it was disrupted. Teachers attributed these missed instructional periods and disruptions to the following: School announcements/assemblies (n = 28) Test preparation activities or testing (n = 27) Student activities (e.g., holiday activities, play practice, field trips) (n = 19) Another teacher came and taught another lesson at scheduled time (n = 14) Time constraints/scheduling conflicts (n = 11) Snow days (n = 4) Fire drill (n = 3) In the spring interviews, several teachers commented that periods were often missed in the spring due to a district emphasis on test preparation for state standardized tests. In some classrooms, teachers needed to stop or delay their implementation of the program to prepare for these assessments. Shared Reading Material Use The following section addresses treatment classroom s weekly use of shared reading lesson components, materials, and best practice routines. Evaluation of Pearson s Good Habits, Great Readers Program 81

89 Focus Lesson In Action ESL/ELL support Professional Development: Modeling Pause and Reflect Mini-Lesson Introduce the Teacher Modeling Text Write About It Introduce the Student Reader Celebrations Days of Use per Week Figure F1. Weekly use of shared reading lesson components (n = 121). Teacher's Guide Shared Text for Teacher Modeling Planning Guide Student Readers Teacher Resource CD Audio CDs of Student Reader Selections Days of Use per Week Figure F2. Weekly use of shared reading lesson materials (n = 115). Evaluation of Pearson s Good Habits, Great Readers Program 82

90 Think-Pair-Share Using Think-Alouds Repeated Reading Using Sticky Notes Using Reading Journals Teaching Tier-Two Words 22% 45% 62% 62% 78% 74% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure F3. Weekly use of shared reading best practice routines (n = 133). Guided Reading Material Use The following section addresses treatment classroom s weekly use of guided reading lesson components, time spent reading and having conversations about reading during guided reading time. During Reading Prompts ESL/ELL Support Graphic Organizer After Reading Activities and Assessments Before Reading Activities and Lessons Options for Further Instruction: Writing Options for Further Instruction: Mini Lesson Options for Further Instruction: Digging Journal Prompts Days of Use per Week Figure F4. Weekly use of guided reading lesson components (n = 111). Evaluation of Pearson s Good Habits, Great Readers Program 83

91 Table F1 Guided Reading Time Spent Reading and Having Conversations About Reading in Treatment Classrooms Less than 10 minutes minutes minutes minutes More than 40 minutes % # % # % # % # % # Time spent on reading Below-Level Readers 28% 36 35% 45 19% 25 9% 11 9% 12 On-Level Readers 25% 31 35% 43 23% 28 10% 12 8% 10 Advanced Readers 29% 35 29% 35 23% 28 12% 15 7% 9 Time spent having conversations about reading Below-Level Readers 36% 47 39% 51 11% 14 9% 12 5% 6 On-Level Readers 38% 48 37% 46 12% 15 8% 10 5% 6 Advanced Readers 42% 52 31% 38 15% 19 7% 9 5% 6 Evaluation of Pearson s Good Habits, Great Readers Program 84

92 Use of Words their Way and Quick Reads The following section addresses treatment teacher use and perceptions of Words their Way and Quick Reads components. Table F2 Teacher Use and Perceptions of Words Their Way and Quick Reads Percentage who used component Which students received this instruction? How did teachers decide when to use this component? What did teachers like about the component? What didn t teachers like about the component? Words their Way Quick Reads 46% 25% All students: 42% All students: 14% Below-level readers: 8% On-level readers: 4% Suggestions from the planning guide (n = 2) Regularly scheduled time points n/a (n = 2) When there is extra time (n = 2) When teacher noticed a student Enjoyed the sorts (n = 7) I like that I can do a whole class instruction and then do different sorts for the guided reading groups. Patterns are emphasized each week (n = 5) Grouped spelling lists/strategies (n = 5) Wide variety of words (n = 2) Some words or sorts are too difficult and some belowlevel students experience difficulty (n = 8) Definitions are not included in the teacher s guide (n = 3) Vocabulary is not applicable to students lives (n = 2) was struggling (n = 2) Engaging/interesting topics that maintain student attention (n = 3) Usefulness of the component as an assessment tool (n = 2) Ability to show students different strategies (n = 2) Computer or technical difficulties (e.g., software does not work on their computers, not enough computers) (n = 6) Classroom management issues (n = 3) What were teacher suggestions for improvement? Include more activities per lesson (n = 9) Integrate activity with Smart boards (n = 3) n/a Evaluation of Pearson s Good Habits, Great Readers Program 85

93 Writing Material Use The following section addresses treatment teacher s weekly use of writing lesson components and writing trait time lesson components. Figure F5. Weekly use of components (n = 95). Apply Lesson Teach writing lesson ESL/ELL Tips 3.07 Grammar/Usage/Mechanics mini-lesson 2.9 Figure F6. Weekly use of writing trait time Share 2.75 ESL/ELL Tips 3.2 lessons (n = 51). Conference Prompts Grammar/Usage/Mechanics Mini-Lesson Introduce Share Make it Better Evaluate Models Days of Use per Week 2.31 Define Days of Use per Week Evaluation of Pearson s Good Habits, Great Readers Program 86

94 Supplemental Program Components The following tables address teacher use and perceptions of end-of-unit writing lesson components, professional development pieces, My Notes, and the program online database. Table F3 Teacher Use and Perceptions of End-of-Unit Writing Lesson Components Wrap Up Informal Grammar Follow-Up Wrap Up Publishing Assessment Extension Activities Percentage who used component 46% 32% 25% 13% What did teachers like about the component? Students create a finished product and share it with others (n = 8). Social studies report writing was shared with class at school wide state fair with extremely positive results. Good way of practicing what they learned over the week and tying everything together (n = 7). Gives students a chance to reflect on their accomplishments (n = 6). I like giving the time to celebrate and publish our work. The students really enjoy this time to reflect. Allows for simple or quick evaluation (n = 8). Allows for reflection on self as a teacher (n = 5). Helps with differentiated instruction (n = 5). Good reinforcement of skills (n = 4). I liked that certain areas of grammar are brought to the student s attention and they can use it in their writing. Extra practice or review (n = 3). Can be done quickly (n = 4). Digging Deeper Extension Activities Good for advanced group (n = 4). Students that have mastered the concept can take what they learned further. Encourages students to think at a higher level or to analyze (n = 2). Use as a type of assessment (n = 2). Evaluation of Pearson s Good Habits, Great Readers Program 87

95 What didn t teachers like about the component? Needed to supplement material due to school or state requirements (n = 1). In reality, it takes longer to publish (n = 1). Some instruction feels redundant (n = 1). One teacher commented that the ideas were just okay. Some lessons are too difficult for some students (e.g., special education) or are not student friendly (n = 3). n/a Evaluation of Pearson s Good Habits, Great Readers Program 88

96 Table F4 Teacher Use and Perceptions of Good Habits, Great Readers Professional Development, My Notes, and Online Database Percentage who used component Additional Information Professional Development My Notes Online Database 42% 29% 17% Ninety-six percent of teachers who used the component found it to be somewhat or very useful. Teachers reported using the 4-page professional development pieces to help with information to incorporate into lesson plans (n = 12) and as a way to acquire background knowledge on the content (n = 10). Additionally, seven teachers used the professional development pieces to better understand the program or their students, three teachers used it for self-reflection or growth purposes, and two used it as a review. Two teachers who did not find the professional development piece useful cited time factors and a redundancy of information. The majority of teachers who did not use the My Notes section noted that they have other methods to reflect on the unit (e.g., can do this informally, write in their own planning book) (n = 19) or that they did not find it necessary to use this component (n = 17). Several other teachers mentioned a lack of time (n = 9) or not thinking about the component (n = 5). Finally, two teachers wrote they did not have access to their own teacher s edition (n = 2) or did not want to write in the manual (n = 2). Teachers reported using the database to access lesson plans at a lower level (n = 17) and to access lesson plans at a higher level (n = 11). Fifty-five percent of teachers found some of what they needed through the database, and 45% found all of what they needed. Teachers frequently reported searching the database by: DRA level (n = 12) Skill level (n = 11) Guided Reading level (n = 9) Grade Level (n = 2) Before participating in the study, 31% of teachers had accessed the online database. Teachers who did not use the online database during the study noted that they did not need to access the online database, because they obtain higher- or lower-level materials from other sources (47%), did not need materials at higher or lower levels (26%), or other reasons (21%). Frequent other reasons for not using the online database included having enough materials or using some other source (n = 4), problems with their computer (n = 3), not having enough time (n = 3), or not knowing about the resource (n = 2). Evaluation of Pearson s Good Habits, Great Readers Program

97 Guided Reading Material Use by Control Teachers The following table addresses control classroom s time spent reading and having conversations about reading during guided reading time. Table F5 Guided Reading Time Spent Reading and Having Conversations About Reading in Comparison Classrooms Less than 10 minutes minutes minutes minutes More than 40 minutes % # % # % # % # % # Time spent on reading Below-Level Readers 46% 6 46% 6 8% 1 On-Level Readers 54% 7 31% 4 15% 2 Advanced Readers 8% 1 39% 5 39% 5 15% 2 Time spent having conversations about reading Below-Level Readers 31% 4 46% 6 15% 2 8% 1 On-Level Readers 23% 3 46% 6 31% 4 Advanced Readers 31% 4 31% 4 39% 5 Evaluation of Pearson s Good Habits, Great Readers Program

98 APPENDIX G Supporting Tables for Student Performance Results The tables below present the results of multilevel model analyses examining Good Habits, Great Readers program participants pretest to posttest gains on the GMRT-4. Readers should use caution when interpreting these results because the analyses divide the sample. Evaluators calculated effect sizes to provide insight regarding the practical importance of these findings. Table G1 Multilevel Modeling Results Examining GMRT-4 Vocabulary Gains for Subgroups of Treatment Students Student Subgroup Coefficient Standard Error t- value Approx. df p- value Effect Size Male (vs. female) Caucasian (vs. non-caucasian) FRL status (vs. non-frl status) LEP (vs. non-lep) * Special education vs. non-special education Fifth grade (vs. fourth grade) * Significant at the.05 level Percentile Gain * * Table G2 Multilevel Modeling Results Examining GMRT-4 Comprehension Gains for Subgroups of Treatment Students Student Subgroup Coefficient Standard Error t- value Approx. df p- value Effect Size Male (vs. female) Caucasian (vs. non-caucasian) FRL status (vs. non-frl status) LEP (vs. non-lep) * Special education vs. non-special education Fifth grade (vs. fourth grade) * Significant at the.05 level Percentile Gain Evaluation of Pearson s Good Habits, Great Readers Program

99 Table G3 Multilevel Modeling Results Examining GMRT-4 Total Gains for Subgroups of Treatment Students Student Subgroup Coefficient Standard Error t- value Approx. df p- value Effect Size Percentile Gain Male (vs. female) Caucasian (vs. non-caucasian) FRL status (vs. non-frl status) LEP (vs. non-lep) * Special education vs. non-special * education Fifth grade (vs. fourth grade) * * Significant at the.05 level The tables below present the results of multilevel model analyses examining study condition by subgroup interactions on GMRT-4 gains. Readers should use caution when interpreting these results because the analyses divide the sample. Evaluators calculated effect sizes to provide insight regarding the practical importance of these findings. Table G4 Multilevel Modeling Results Examining the Effects of Study Condition (Treatment or Comparison) by Subgroup Interactions on GMRT-4 Vocabulary Gains Student Subgroup Coefficient Standard Error t- value Approx. df p- value Male (vs. female) * condition * Caucasian (vs. non-caucasian) * condition FRL status (vs. non-frl status) * condition LEP (vs. non-lep) * condition * Special education (vs. non-special education) * condition * Fifth grade (vs. fourth grade) * condition * Significant at the.05 level Evaluation of Pearson s Good Habits, Great Readers Program

100 Table G5 Multilevel Modeling Results Examining the Effects of Study Condition (Treatment or Comparison) by Subgroup Interactions on GMRT-4 Comprehension Gains Student Subgroup Coefficient Standard Error t- value Approx. df p- value Male (vs. female) * condition Caucasian (vs. non-caucasian) * condition FRL status (vs. non-frl status) * condition LEP (vs. non-lep) * condition * Special education (vs. non-special education) * condition * Fifth grade (vs. fourth grade) * condition * Significant at the.05 level Table G6 Multilevel Modeling Results Examining the Effects of Study Condition (Treatment or Comparison) by Subgroup Interactions on GMRT-4 Total Gains Student Subgroup Coefficient Standard Error t- value Approx. df p- value Male (vs. female) * condition * Caucasian (vs. non-caucasian) * condition FRL status (vs. non-frl status) * condition LEP (vs. non-lep) * condition * Special education (vs. non-special education) * condition * Fifth grade (vs. fourth grade) * condition * Significant at the.05 level Evaluation of Pearson s Good Habits, Great Readers Program

101 The figures below present the treatment- and comparison-group means for various ERAS questions. About reading different kinds of books About going to a bookstore About reading instead of playing About reading during summer vacation About starting a new book About spending free time reading a book About getting a book for a present About reading for fun at home When you read a book in school during free time When you read a book on a rainy Saturday Pretest Posttest 29% 27% 45% 39% 61% 53% 67% 62% 63% 57% 60% 66% 66% 72% 80% 79% 88% 83% 90% 85% 0% 20% 40% 60% 80% 100% Figure G1. Percentage of Good Habits, Great Readers program participants who indicated that they felt extremely positive about various situations involving recreational reading. Evaluation of Pearson s Good Habits, Great Readers Program

102 About reading different kinds of books About going to a bookstore About reading instead of playing About reading during summer vacation About starting a new book About spending free time reading a book About getting a book for a present About reading for fun at home When you read a book in school during free time When you read a book on a rainy Saturday Pretest Posttest 26% 20% 35% 30% 84% 86% 83% 81% 87% 88% 61% 50% 61% 54% 61% 57% 72% 64% 61% 59% 0% 20% 40% 60% 80% 100% Figure G2. Percentage of comparison-group participants who indicated that they felt extremely positive about various situations involving recreational reading. Evaluation of Pearson s Good Habits, Great Readers Program

103 About taking a reading test About using a dictionary When you read out loud in class About stories you read in reading class When it's time for reading in class About learning from a book About reading your school books About reading in school About reading workbook pages and worksheets When a teacher asks you questions about what you read Pretest Posttest 43% 41% 57% 53% 54% 56% 76% 69% 74% 66% 78% 76% 61% 54% 84% 75% 42% 40% 62% 54% 0% 20% 40% 60% 80% 100% Figure G3. Percentage of Good Habits, Great Readers program participants who indicated that they felt extremely positive about various situations involving academic reading. Evaluation of Pearson s Good Habits, Great Readers Program

104 About taking a reading test Pretest Posttest 58% 49% About using a dictionary 55% 50% 0% 0% 0% 0% 0% 1 1 When you read out loud in class About stories you read in reading class 52% 55% 78% 80% When it's time for reading in class 75% 75% About learning from a book 85% 84% About reading your school books 63% 68% About reading in school 83% 80% About reading workbook pages and worksheets 43% 40% When a teacher asks you questions about what you read 63% 60% 0% 20% 40% 60% 80% 100% Figure G4. Percentage of comparison-group participants who indicated that they felt extremely positive about various situations involving academic reading. Evaluation of Pearson s Good Habits, Great Readers Program