Utilization-Focused Program Evaluation Title I Reading Program. FOR The Kearney R-I School District Title I Reading Program

Transcription

1 Utilization-Focused Program Evaluation Title I Reading Program FOR The Kearney R-I School District Title I Reading Program Dr. William Nicely Superintendent of Schools Jacque Banks Assistant Superintendent of Federal Programs

2 Table of Contents Utilization-Focused Evaluation Program Evaluation Goals... p. 3 Kearney R-I School District UFE Data Collection, Findings, & Analysis. p. 3 Observations.. p. 6 Interviews p. 14 Survey(s).. p. 26 Recommendations. p. 26 Reference List. p. 33 Appendix A: Plan of Work... p. 35 Appendix B: Protocols and Checklists. p. 38 2

3 Utilization-Focused Program Evaluation Goals A utilization-focused evaluation (UFE) of the Kearney R-I Title I Reading Program was conducted, as commissioned by Dr. William Nicely, Superintendent, and Ms. Jacque Banks, Director of Federal Programs. A Plan of Work was developed collaboratively between Patron Insight, Inc. and the District s executive leadership team (See Appendix A). The overarching goal of the program evaluation was to determine the fidelity of practice in the Title I Reading Program, specifically in grades kindergarten, first, and second in the following areas: Research-based instructional strategies, using the Five Components (Sub-Skills) of Reading Evidence of implementation of reading block Fidelity of implementation of literacy curriculum and the text resource series, Literacy by Design (Harcourt) Fidelity of implementation of the Response to Intervention (RtI) program Notations of other literacy strategies and activities Tools utilized included observations, interviews, compiling and coding results for consistency, reporting findings, and making recommendations based upon findings and analysis (See Appendix B). Kearney R-I School District Utilization-Focused Evaluation Data Collection, Findings, and Analysis Data Collection Elementary school visits. Two visits (one announced and one unannounced) were made to the four elementary schools for the purpose of observing instruction. Incidents were recorded and observations made, including looking for school and classroom culture, 3

4 student engagement, and the level of fidelity of practice utilizing the Five Components (Sub-Skills) of Reading Instruction; evidence of reading block; evidence of curriculum implementation and use of the text resource, Literacy by Design; level of fidelity of Response to Intervention (RtI); and notations about other instructional practices and literacy strategies. Personnel. More than 50 personnel directly or indirectly informed the UFE as participants in observations and interviews: Assistant Superintendent of Special Programs Director of Curriculum, Assessment, Professional Development 4 Elementary Principals 2 Reading Interventionists 4 Remedial Reading Teachers 42 Instructional Staff (classrooms, computer labs, library, paraprofessionals) Professional Development Trainer Survey Respondents District assessment tools. Since assessment drives instruction, it was noted that the District-approved Assessment Calendar guides assessments throughout the four elementary schools. Besides the Missouri Assessment Program test, the Kearney R-I School District utilized other formal assessments focused on reading achievement at the elementary level. Although no data from these assessments were used, they are listed below: K-5 Pearson Developmental Reading Assessment (DRA) K-5 Renaissance Learning STAR Reading Assessment Reading Curriculum-Based Assessment (AIMSweb) MAZE Rigby READS Diagnostic Assessment (LBD) Others 4

5 An additional tool is Pearson AIMSweb, a benchmarking and progress and assessment monitoring tool. K-5 reading data were compiled in AIMSweb to provide feedback regarding student achievement in the area of reading and to inform decisions about intervention and instruction in reading for elementary students placement in Tiers as part of the District s Response to Intervention (RtI) program. Buffum, Mattos, and Weber (2010) suggested, schools should provide targeted and systematic interventions to all students as soon as they demonstrate the need (p.10). In addition, the District Assessment Calendar provided a pacing guide to ensure consistency and accountability for monitoring reading progress for each student throughout the four elementary schools. On-site data collection. The CSIP Goal that guided this evaluation was that 90% of students will read at grade level by third grade. The following topics became the purpose statements for the UFE and informed all observations, interviews, and survey questions. Throughout the report, the order of reporting will be consideration of: Five components, reading block, curriculum and Literacy by Design, Response to Intervention, and other notations about literacy instruction. During seven days of observations and interviews in the Kearney R-I School District at four elementary schools, the following specific components of the reading program were prioritized: Fidelity of research-based instructional strategies using the Five Components (Sub-Skills) of Reading Evidence of implementation of reading block Fidelity of implementation of literacy curriculum and the text resource series, Literacy by Design Fidelity of implementation of the Response to Intervention (RtI) program Notations of other literacy strategies and activities As stated in Kearney R-I Literacy Look-For Checklist, observations are a snapshot of classroom instruction conducted during a portion of each teacher s reading block. It is not expected to see all of the following components in every classroom 5

6 observed during this observation. This statement set the stage for Findings, Analysis, and Recommendations generated from data reported from observations for this UFE of Title I Reading (See Appendix B). General themes emerged from respondents during observations and interviews with the use of evaluation tools, such as checklists and protocols. Findings from Observations Student Learning Observations. Observations included walk-throughs to observe teaching and learning in 39 classrooms K-5. Additional observations included a second visit to some classrooms and observations of Tier 3 intervention during RtI conducted by paraprofessionals and other staff assigned to groups of students. Visit one was to observe implementation of the Five Components and to ascertain the level of student engagement, culture of learning, and general implementation of reading instruction, while visit two was specific to a time scheduled for Response to Intervention. Visits included observations in four elementary schools, 39 classrooms with a total of 54 observations K-5. o School 1 K-2 7 Observations 3 Additional Observations o School 2 K-3 9 Observations 1 Additional Observations o School 3 K-2 10 Observations 6 Additional Observations o School 4 K-5 13 Observations 5 Additional Observations Results from the observations are presented below and include the number of incidents observed for the categories noted in Figure 1, as well as a general observation of other literacy or school routine activities during school visits. A percent of incidents observed during this evaluation was generated and noted. Figure 1 provided observation data for reading block, explicit instruction of the Five Components of reading, and a listing of other observations, as noted during visits. Of the observations made specifically to note implementation of the Five 6

7 Components and reading block, 36% of the classrooms were implementing some component of reading block. In classrooms where the Five Components were observed, strategies were implemented with fidelity, and students were highly engaged in learning to read and apply strategies. See Figure 1. School Visit 1 School 1 Visit 2 School 1 Visit 1 School 2 Visit 2 School 2 Visit 1 School 3 Visit 2 School 3 Visit 1 School 4 Visit 2 School 4 Reading Block Phonics Phonemic Awareness Vocab Fluency Comprehension Other Activities Daily 5 Morning Work Writing Daily Writing (1 in Gr. 3) Daily Spelling Morning work Data Team Writing Daily Daily 5 Writing Computer Centers Computers Library Total n 18/50 23/50 24/50 3/50 13/50 11/50 % 36% 46% 48% 6% 26% 22% Figure 1. Incidents by Reading Category during observations for K-2 during two visits. Analysis of Observations Five Components (Sub-Skills) of Reading. The Five Components were implemented with moderate fidelity and consistency, given the level of implementation that varied throughout the district. Of the observations in which teachers were directly and explicitly using the Five Components of Reading Instruction, all four schools were utilizing the Five Components with fidelity and consistency in K-1-2 classrooms. The frequency of explicit instruction was measured only by the opportunity for observation, as denoted in Figure 1. However, observation of anchor charts with phonetic cues, listening to students reading aloud and self-correcting, as well as brief conversations 7

8 with teachers revealed that a focus on the Five Components of Reading was prioritized at all schools in all K-1-2 classrooms as well as in remedial reading classrooms and during RtI. Observations revealed that the majority of explicit instruction was primarily using two components (phonics and phonemic awareness). It is notable that most of that explicit instruction was in a whole-group setting. Seldom was explicit instruction for vocabulary, fluency, and comprehension strategies observed. Reading block. Observations during 50 visits (K-5) revealed a total of 17 classrooms, or 36%, were implementing a component of reading block (explicit whole-group instruction, small-group instruction in specific skills, guided reading, shared reading, paired reading, writing, vocabulary word work, or other reading activities). During two visits to each of the four elementary schools, 100%, or all schools, had at least one example of implementation of reading block. Observations yielded information that teachers used flexible groups that rotated through centers, providing indirect literacybased instruction (e.g., Study Island, word-works, read to self, and other attributes of the Daily 5). Explicit instruction observed during reading block sessions included work primarily focused on phonics, phonemic awareness, and fluency. There was limited observation of instruction in vocabulary, fluency, and comprehension. The use and routine of the Daily 5 strategy was moderate to high in use and was posted in several of the classrooms. Due to limited observations and the low number recorded, the level or fidelity of implementation of reading block cannot be determined. However, components of reading block were evident. In addition, observations and conversations revealed that setting aside a block of time for reading instruction and/or literacy activities was consistent for all schools at some time during the day. Although it was evident in all four schools, it was undetermined in some schools as to what time a reading block was held or to the level of consistency it was as an instructional model. Curriculum. Implementation of the District s English Language Arts (ELA) curriculum was not observed. Written objectives were not observed but verbal statements of 8

9 classroom activities and expectations were 100% evident (Daily 5, class routines, purchased posters, and some teacher- or student-generated anchor charts). It was clear that students understood the routines and expectations of learning, including morning routines, Daily 5, movement in and out of centers, as well as shifts from one activity to another. Most important, students were always engaged in a learning activity in every classroom observed. With regard to the reading curriculum, the data supported that the Five Components were a curricular focus and used as instructional strategies. Teachers were utilizing resources to navigate students through routines of rhyming words, nonsense words, and phonemic segmentation. Although there was a clear expectation of engagement and a culture of learning, there were no data to report evidence of use of the District s approved curriculum. It should be noted that the English Language Arts curriculum was being revised. Literacy by Design (2008, Harcourt Achieve, Inc.). There was moderate fidelity of use for the District s approved text resource, Literacy by Design. During observations and as a probe during interviews, it was with randomness that the text resource was in use. Observations did not provide opportunity to ascertain the text source for writing lessons, RtI Tiers 1, 2, or 3 instruction, or texts utilized during each classroom observation. To summarize, note the following statements from the extent of observations in which LBD was evident and utilized: One teacher shared information about upcoming lesson plans in which a unit would be utilized from LBD. Instruction was not observed. K-1 teachers used flipbooks (phonics and phonemic awareness instruction), which may have been a LBD resource. One teacher utilized a Big Book, whole-group, non-fiction vocabulary and comprehension lesson that was rich in student engagement. During classroom walk-throughs, LBD fiction and nonfiction texts as well as some science and social studies nonfiction texts were evident in classroom libraries. Students read LBD texts from classroom libraries. 9

10 Response to Intervention. The importance of establishing a discussion of RtI is best presented from Buffam, Mattos, and Weber (2006) who suggested that RtI is a research-based practice that targets specific learning needs students have. Because learning styles and instructional needs vary from student to student, we must provide each student with targeted instruction that is, teaching practices designed to meet his or her individual learning needs. We also know that students don t all learn at the same speed. Some will need more time to learn. That is the purpose of RTI to systematically provide every student with the additional time and support needed to learn at high levels (p. 10). For the purpose of observations during this evaluation, the definition of Response to Intervention was a combination of the District s expectation to implement coupled with a clinical definition that is embodied in a quote from Buffum, Mattos, and Weber (2006):... schools should provide targeted and systematic interventions to all students as soon as they demonstrate the need (p. 10). Kemp and Eaton suggested, RtI is best depicted as a model that incorporates the use of a tiered system, which focuses on accountability and academic supports (cited by International Reading Association, , p. 4). The District s expectation was that during a 30-minute period of time, preferably during the first 30 minutes of instruction each day, teachers were explicitly instructing reading skills targeted toward specific needs of students in whole-group, small-group, or individual instruction. Students should be engaged in learning specific reading skills determined by individual assessment results and placement in one of three Tiers. Tier 1 included whole-group instruction and/or small-group and individual reading opportunities. Tier 2 students received explicit reading instruction, focused on individual learning needs in a small group. Tier 3 students received explicit reading instruction focused on individual learning needs in a small group or in a one-to-one model. District expectations for RtI included explicit instruction using research-based skills (e.g., the Five Components of Reading) and should be conducted in a dedicated period of time separate from reading block and other literacy activities throughout the 10

11 day. The dedicated 30 minutes of explicit reading instruction during RtI were to ensure each learner s needs was met. Based on a problem-solving model, the RTI approach considers environmental factors as they might apply to an individual student s difficulty, and provides services/intervention as soon as the student demonstrates a need (PBIS, 2014). RtI at schools. A notable observation was that three schools were deliberate in scheduling RtI with a prescribed time for RtI at the beginning of the day, while one school had a random schedule with each grade level or classroom scheduling RtI time around other prescribed school activities or classes. All four elementary schools appeared to have a unique approach for the RtI session. One school used Tier 1 for enrichment opportunities, Tier 2 for supporting reading needs, and Tier 3 for explicit reading skills instruction. As observed, other schools had primarily whole-group instruction or Tier 2 activities, such as the Daily 5 with some Tier 3 pull-out instruction. There was a random approach for RtI implementation among the four elementary schools and, in some cases, by classroom. Therefore, based on observations in schools two times during March, RtI, as defined in the research, was implemented with moderate fidelity, depending on the school and the classroom. Figure 2 provides data describing the number of observations specifically to determine the consistency of implementation of a required Response to Intervention program. The second observation visit to kindergarten, first-, and second-grade classrooms was unannounced and was based on the RtI schedule provided by each school. Determination for implementation was focused on only the definition of RtI provided previously. If classrooms were involved in spelling activities, morning work, Daily 5 or other literacy activities, it was noted but not analyzed. Data revealed the number of classrooms observed and the number of classrooms in which RtI was observed. A percent for the observations was generated. Figure 2 depicts the data that reveal the level of consistency and integrity of practice in implementing RtI during a prescribed time set aside in four elementary schools. 11

12 School Classrooms Visited Observed RTI Percent Implementation Implementation # % # % # % # % Totals % Notations: minutes/school looking for explicit reading instruction during RtI. Includes Tier 3 pull-out as observed but does not include writing spelling, morning work, or non-reading activities. All students were engaged in learning during the scheduled period of time; however, explicit reading instruction during RtI was observed at a level of 67%. Figure 2. Implementation of RtI at four elementary schools during a scheduled time period provided to the evaluator during two separate observations. Figure 3 provides data describing the number of observations specifically to determine the level of consistency in implementing the Five Components of Reading during RtI at four elementary schools. Students were instructed during whole-group instruction and small-group instruction in which they were engaged in learning targeted reading skills, usually phonics or phonemic awareness with a limited number in fluency and comprehension and none in vocabulary. Incidents of students pulled out for Tier 3 intervention throughout the day were not observed. School & # Incidents Tier 1 Tier 2 Tier 3 Notations re: Implementation #1 2/ K moved from T1 to T2 (2 incidents) #2 3/ Includes para pull-out #3 4/6 1 3 Not observed K moved from T1 to T2 (2 incidents) #4 7/ K T2 & T3 counted as 2 incidents Total 16/ % observed 21% 33% 21% Figure 3. Number of incidents of targeted instruction using the Five Components during the prescribed time of day for RtI during an unannounced observation in K-2 classrooms. 12

13 Notations for other literacy activities. Figures denoted observations of the Five Components as the research-based, sub-skill focus for explicit reading instruction. Direct or indirect instruction using other strategies was not recorded for consistency but was noted during visits to kindergarten, first-, and second-grade classrooms. Additional observations of literacy intervention, instruction, or learning activities in K-5 classrooms included: A Data Team meeting focused on academic and social/emotional intervention Tier 3 reading instruction with a reading teacher 5th grade science fair presentations 5th grade social studies pairs using tools for content area reading 4th grade writing instruction 3rd grade reading block 2nd grade reading block Reading teacher push-in for Tier 2 fluency and comprehension instruction during reading block 2nd grade writing and language arts instruction (2) Kindergarten and 1st grade journal writing with prompts Student read aloud Activities preparing for spelling tests Library literature activity in preparation for an author s visit Computer lab activities in math and literacy (reading, writing, word work and math) and ipad activities using Study Island 3 whole-group gatherings in which the teacher read aloud to students Big Book science reading comprehension/vocabulary conversations, checking for understanding Engagement and classroom expectations. In addition to the data already reported, some reading block activities, Daily 5, spelling, journal writing prompts, morning work and routines, computer lab visits, writing instruction, spelling test preparation, early morning gathering activities, whole-group Big Book reading, and other school-related 13

14 activities were evident during both visits during scheduled RtI time and scheduled reading block. As observed, all students in every classroom were engaged in learning activities or routines directed by the teacher. Centers were the norm in every classroom. The culture of learning and expectations for student engagement were clearly set and implemented in the Kearney R-I School District s elementary classrooms. Findings from Interview Sessions Interviews. Most interviews were conducted on-site, during scheduled times with the respondents. Two phone conferences and some correspondence provided background information, data results, archival data, and clarification as needed. Two questions guided all interviews: What is working? and What can be better? The responses were coded and general statements were derived from responses. The following personnel were interviewed during scheduled visits or by phone or . 2 Central Office Administrators 4 Elementary Principals 2 Reading Interventionists 4 Elementary Remedial Reading Teachers 1 Professional Development Trainer Questions. First, two questions were asked of 11 respondents: What s working? and What can be better? Throughout the interviews, and because the protocol focused the conversation as such, these two questions became the organizational process by which the topics were discussed and reported. To that end, these guiding questions provided a framework to identify strengths and weaknesses of UFE goals and identified programmatic topics related to reading instruction. Principal ratings for level of implementation. During interviews, four elementary principals were asked to rate the level of implementation from 1 to 10 for eight instructional areas related to the overall reading program K-5. The possible total was 40 (the average of a range from 1-10 for each topic from four elementary principals). A 14

15 total for each instructional area was determined and a percent generated. Principals supervise teaching and learning on-site and would have the most exposure to rate the level of implementation of District initiatives. Bold indicates low and high. Italicized indicates low. See Figure 4. Implementation K-2 Percent Range 3-5 Percent Range Five Components 75% % 5-6 Reading Instruction 80% % 5-7 Curriculum 80% % 5-9 LBD Resource 60% % 3-9 RtI 77.5% % 4-7 Best Practices 77.5% % 5-7 Use of Data 72.5% % 6-9 Other Literacy 72.5% % 4-6 Figure 4. Principals ratings in response to rating the level of (fidelity) integrity and continuity of these strategies or initiatives (Scale of 1-10). Principals used a wide range in rating use of LBD resources, indicating implementation in some pockets and very little in others. K-2 ratings were consistently higher in all categories except in use of data. The highest ratings were for reading instruction and curriculum in K-2 at 80%. The lowest rating was 52.5% for other literacy instruction that included writing, speaking, listening and thinking in grades 3-5. Findings for What s working? Five Components. Unanimously, all respondents stated that the professional development for the Five Components has been instrumental in the current level of implementation and in targeting students learning needs. One respondent stated: I didn t know what I didn t know. Comments from interviews were synthesized into general statements: The focus and message that set a common goal has increased implementation more quickly in grades K-2. 15

16 Some teachers learned, implemented, and continue to make progress. Teachers are open to implementing the Five Components and are perfecting the use of the tools as the year continues. Reading teachers are fully and consistently implementing the Five Components. The level of fidelity at K-1-2 is higher and improving, especially for Year 1. The Five Components provide a common language and common strategies. There is a consistent direction for instruction in all schools that will impact students and how well they will read. Reading block Teachers are implementing reading block at various levels, but many are trying it out. Implementation varies by school and by grade level. Professional development has assisted in understanding the 90-minute reading block and why reading strategies must be explicitly taught. Reader s block provides another opportunity for explicit teaching and implementation of the Five Components to target student needs. There is growth in implementation of reading block but not strong enough, even though some progress has been made in a short amount of time. There is better understanding regarding flexible groups, and the use of smallgroup instruction is needed in order that students succeed. Implementation of curriculum and text resource, Literacy by Design (LBD) There is a curriculum, but it is being revised. LBD is available to be used. LBD has good components to it. Student resources are helpful, especially in the study of nonfiction topics (science was specifically mentioned). The LBD pacing guide could help guide instruction and promote consistency. LBD will be a helpful resource to implement the 90-minute reading block. 16

17 Strategies such as vocabulary, metacognition, main idea, etc. are stronger components. Lead teachers were trained in using LBD. Response to Intervention (RtI) RtI directly targets an individual student s needs and provides extra instruction. Generally, RtI is a set-aside time for targeted instruction. Tier 3 students are targeted for specific instruction and could receive reading instruction multiple times during a day or week (Tier 1-classroom instruction in reading block, Tier 2-reading block, Tier 3 RtI, and sometimes Tier 3 in special reading). Most teachers are implementing in some way, especially K-2. RtI is supporting classroom reading instruction, especially if using the reading block model. Monitoring individual students and determining the appropriate Tier enables immediate monitoring of changes in students skill levels (AIMSweb). Interventionists Interventionists provide support for implementation of strategies, specifically the Five Components and implementation of reading block. They provide assistance with data monitoring. They are a resource for teachers with questions and concerns about implementation. They build trust and are trusted and are part of staff data teams. They help bridge learning for teachers between professional development sessions. Interventionists are a great addition and are phenomenal. 17

18 Assessments Multiple assessments are used to monitor progress of individual students. o DRA o R-CBM o Running Records o STAR o Limited mention of LBD assessment (Rigby READs) Data sources are used to analyze and specifically target instruction to meet each student s needs. Professional development The professional development consultant delivers and follows through. Currently, PD is consistent and focused. PD is presented so that it can be implemented. The information is research-based and usable. Findings for What can be better? Five Components All schools should implement the Five Components with the same expectations. The Five Components should be ongoing, consistent, and monitored for implementation in RtI, reading block, and special reading. K-1-2 teachers should implement the Five Components with fidelity. Prepare all teachers for implementation by providing a plan and level of expectation. Teachers, interventionists, and principals need to observe best practices of implementation at other schools. Stay focused on the Five Components until the strategy is fully implemented with fidelity in all K-5 classroom. 18

19 Be sure there is full understanding of how to explicitly teach the Five Components of reading (not just an overview) in whole-group, small-group, and individual settings. Focus instruction on more than phonics and phonemic awareness (include more targeted instruction in fluency, vocabulary, and comprehension). Continue training until everyone understands instruction in all components and with full implementation. Bring new teachers along immediately. Monitor implementation to ensure fidelity. Reading block Design an implementation plan consistent with expectations. At implementation, expectations should be all schools all in or as the plan prescribes. Observing high-quality reading blocks at other schools will provide a model before full implementation. Provide professional development as part of implementation. Set high expectations for implementation and monitor for integrity of practice and consistency of implementation. Explain that reading block is more than breaking into groups. Use the pacing plan in LBD. Provide time for full implementation before adding other expectations. Identify whether or not writing is a part of the 90-minute block. Observe effective models or the 90-minute reading block in district and out of district (principals, classroom teachers, and reading teachers together). Re: push in: Before implementing a push-in model, key staff (principals, classroom teachers, and reading teachers) should observe classrooms where push-in is working. Set expectations for push-in and pull-out (e.g., opt-in parameters, number of times per week, and role clarification of reading teachers). Train and pilot before implementing. 19

20 If push-in is the expectation, work on scheduling models for better efficiency. Classroom teachers should be trained in how to team with reading teachers effectively. Reading teachers should be trained in how to team with the classroom teacher and still meet the needs of special reading students (co-teaching or team teaching models). Consider how caseloads for special reading teachers will be impacted. Curriculum and Literacy by Design (LBD) Note: There was no discussion of curriculum from respondents, except with regard to the priority of the Five Components, use of the Daily 5, journal writing, and spelling as areas of instruction and that the ELA curriculum was being revised. There was some discussion regarding curriculum alignment across grade levels (lateral and vertical alignment). Define the expectations for using LBD (all schools all grade levels). If using LBD as a primary text resource, supplement weaker areas of LBD (varying ideas about weak areas). Designate the LBD pacing plan as the guide for reading block. Set expectations for consistent use of LBD (all schools all in). Some LBD components are not as strong as needed (e. g., fluency). If LBD or another resource is used, train teachers and administrators to implement with consistency and with fidelity. Other resources o There is a random collection and no consistency of resources used as supplements throughout the District including: Pathways (fewer than 10 teachers) Fountas and Pinnell Leveled Literacy Intervention (LLI Harcourt) Calkins (writing) Others? 20

21 Response to Intervention Set expectations for implementation of RtI: All schools all in. Determine a time of day and number of minutes for all elementary schools. Training was a few years ago, so a review with staff regarding how RtI will be implemented at all schools in all classrooms will be helpful. Define components of RtI and how to implement them. Expect targeted reading instruction during RtI and provide a model for explicit instruction during RtI. Monitor for full implementation. Review how RtI aligns with reading block and remedial reading. Review Tier 2 intervention strategies and placement of students based on need. RtI is more than group work. Provide models of high-performing classrooms and implementation of all three Tiers. If paraprofessionals, librarians, counselors, and other staff are providing explicit instruction, train them to target needs and use strategies. There are varying levels for implementation of RtI. Interventionists Clarification of the position description for teachers, interventionists and principals will be helpful. The supervisory organizational chart appeared confusing to respondents with regard to who supervises, who evaluates, and what the line of communication is. Interventionists should be called literacy coaches to align with the role they play and needs within the schools. Interventionists should: o coach by modeling lessons, team teaching, and troubleshooting instruction of reading strategies. 21

22 o be the link between the professional development consultant and the classroom to provide training when appropriate. o assist in leading data team sessions. o participate in leadership meetings about their role in future plans. o have a consistent role in each school. Interventionists have become so valuable to the effort to improve reading, there should be one interventionist per elementary school and their title should be changed to reflect the nature of their work that of literacy coach. Assessments All four schools should use the District s Assessment Calendar as provided with monitoring for fidelity. Data should be monitored to ensure the needs of students are addressed. Assessment data should inform and target interventions for individual student needs. Review purpose, use, and expectations of the Assessment Calendar and prescribed assessments. Assessment data should inform data team sessions. Professional development Provide professional development for all staff responsible for implementing a new strategy (e.g., reading block and all others). Stay with a focus until fully implemented (e.g., Five Components). Provide follow-up throughout and monitor implementation until the strategy is the norm. As new teachers are hired, develop a plan to train and monitor them so that students do not experience a gap in implementation. Set expectations and monitor for implementation of all strategies. 22

23 There must be consistency of implementation for any professional development training. Planning meetings for implementation should include staff and administrators. All schools all in should be the expectation when implementing strategies addressed in the professional development plan. Other literacy activities Set expectations for writing instruction and/or writing block. Create a guide that sets expectations for literacy instruction as a whole. Provide time for implementation. Provide opportunities to observe high-quality literacy blocks (within the District and outside the District). Continue implementing the Daily 5. Review and set expectations for use of Study Island as an intervention. Continue to provide resources for classroom libraries (e.g., District grants). Analysis of Interviews Summary analysis. What s working? and What can be better? are two questions that guided 11 interviews. Because the statements in each section above provided narrative discussion regarding strengths and weaknesses of the topics (Five Components, reading block, curriculum/lbd, RtI, and other literacy activities), the summary analysis identifies themes that emerged from all interviews. The Title I Reading Program had the parts and pieces of an effective reading program; however, threading reading strategies into the regular classroom at all levels appeared to be moderately consistent, and there were questions about implementation. Observations revealed many examples of effective teaching and learning, and all students were engaged in some type of learning activity. The Five Components (Reading Sub-Skills) were positively impacting K-2 readers and were the strategies that were most implemented with fidelity, 23

24 especially in kindergarten and first grade. There were varying levels of implementation among the four elementary schools (from trying it to full-on, consistent implementation). Although there was no reticence to implement reading block, there appeared to be some concern about how to implement reading block. Teachers were trying and testing. Response to Intervention was implemented with moderate fidelity. Although there were many pockets of full implementation in some schools, the District as a whole was not meeting the stated goal for implementation of explicit, targeted instruction in K-5 elementary schools as denoted in the CSIP. Literacy by Design (Harcourt) was used with moderate fidelity. There was evidence of some use of collections in classroom libraries and some use of instructional flipbooks and whole-group Big Books. Some principals felt strongly that there was a high level of implementation. The District s approved curriculum was not observed. However, other strategies observed (such as the Five Components) may be considered components of the curriculum as well. There was some evidence of a writing program in the elementary schools. In reading the walls for displays of student work, it was observed that some student writing was posted on walls. Few examples were posted in rooms. Most writing was narrative with a few examples of nonfiction. Fidelity of a writing program was not determined, due to lack of observation. All staff (observed and interviewed) implemented various strategies, but the level of implementation varied as to the effectiveness and fidelity of an aligned, articulated reading/writing/language arts curriculum. The level to which a full literacy experience was present would require observation for longer periods of time. There were pockets of full implementation of literacy instruction. Interviewees preferred to focus on a strategy and stay with it until it was implemented and to continue professional development training until a strategy was fully implemented. 24

25 Respondents repeated the need for explicit expectations and timelines for implementation of strategies at all schools in all classrooms. Monitor implementation and hold all accountable at the same level. Continue conversations about implementation of strategies. Adjust when implementation dips, troubleshoot, and provide additional training. All schools all in became an immediate theme from the first visit until the last interview. Consistent themes of concern emerged regarding: o Process used to implement required strategies or initiatives (Five Components, reading block, LBD, RtI, writing instruction, and required assessments). o Perceived optional use of the Assessment Calendar. o Perceived optional use of LBD as the District s text resource. o Whether or not push-in was required and how to implement it. o Optional scheduling/implementation of RtI. o Varying RtI models and fidelity of implementation across the District. o What strategy was to be implemented and when it was required (e. g., reading block). o Expectations set for K-2 only (and less on grades 3-5). o Random consistency of expectation for implementation of strategies at the four elementary schools. Findings and Analysis of Quick-Response Survey(s) Online surveys. During late March and early April 2014, two, quick-response, online surveys were administered by Patron Insight, Inc. The first survey was sent to all patrons for whom Kearney R-I School District had an address and focused on interests patrons and staff had regarding programs that are implemented in the District. Results from the Interest Survey created areas of interest from which the second, quick-response online survey emerged. There were 504 respondents who 25

26 completed the initial Interest Survey and from those responses 151 indicated they would participate in a survey focused on the Kearney R-I Title I Reading/Remedial Program. In addition, the Assistant Superintendent for Special Programs added the Kearney R-I staff as opt-in participants. Findings and an analysis from the survey responses will be analyzed and presented in an addendum document provided by Patron Insight, Inc. Recommendations for the Kearney R-I Title I Reading UFE In spring 2014, the Kearney R-I School District in Kearney, Missouri, engaged Patron Insight, Inc. for the purpose of conducting a utilization-focused program evaluation of the Title I Reading Program in four elementary schools: Kearney Elementary, Dogwood Elementary, Hawthorne Elementary, and Southview Elementary. Recommendations for the Kearney R-I Title I Reading Program were the result of moment-in-time observations during seven days, 60- to 90-minute interviews with key staff members, and results from the Title I Reading Program survey (addendum will be provided). Recommendations were based on qualitative findings as well as the counting of incidents when possible to determine the fidelity of practice in the purpose statements set forth by the Kearney R-I District Administrators. Those purpose statements were to determine the: Fidelity of research-based instructional strategies using the Five Components (Sub-Skills) of Reading Evidence of implementation of reading block Fidelity of implementation of literacy curriculum and the text resource series, Literacy by Design (Harcourt) Fidelity of implementation of the Response to Intervention (RtI) program Notations of other literacy strategies and activities Leaders matter (Sparks, 2005, p. 5). In the case of the Kearney R-I School District and the Title I Reading Program, leadership was intensely focused on developing a plan to 26

27 reach a Comprehensive School Improvement (CSIP) Goal to ensure that 90% of students will read at grade level by third grade. Strategies to reach the goal were developed, training was ongoing, and implementation was evident in pockets throughout the four elementary schools. School leadership played a role in the fruition of the goal. Specifically, elementary principals, interventionists, grade-level team leaders, and all teaching staff may be part of the leadership to ensure the goal is met. Ensuring 90% of third-grade students are reading on grade level is an arduous and challenging goal that will encourage commitment to a theme of all schools all in. Implementation. Since the CSIP Goal and Strategies were set by Kearney R-I School District, the direction was clear. An implementation plan may include: A detailed plan to set direction in order to reach the goal Intensely focused and ongoing professional development for all staff and a plan to bring on new teachers each year Mapping (curriculum, resource references, timelines, assessment checkpoints, monitoring of implementation and results) Tools for implementation Expectations for implementation Time to implement with fidelity Evaluation components for implementation and continued training and monitoring Alignment of the plan for a systematic introduction, training, and implementation of strategies Consistency of continuing a strategy and clear communication as to how it fits into the whole plan, including alignment with past initiatives Commitment to institutionalize the strategy make it the norm A system s approach to implementation. 27

28 In the case of Kearney R-I School District s Title I Reading Program, although this program evaluation revealed some areas of moderate implementation, all of the essential parts of the puzzle were in place including: Caring, professional staff who clearly wanted to do well in putting all of the parts together into a whole, streamlined instructional program Research-based/evidence-based strategies that were proven to improve student performance in reading A professional development plan that provided direction and training An intervention program that involved and had potential to meet the academic needs of 100% of elementary students An assessment plan that constantly produced data to inform reading instruction, monitored results, and targeted instruction that intervenes for individual students Intense focus and available resources Leadership that was in place to monitor implementation Stamina, intensity, and purpose to meet the academic needs of children Recommendations for use of Five Components. Observations of the Five Components indicated this strategy was the most implemented with fidelity in grades K-2. The District may choose to continue implementation of the five sub-skills by providing ongoing training with the same professional development consultant. Communicating the CSIP goal regularly and emphasizing the importance of why the Five Components were chosen as a strategy will reinforce the CSIP Goal and Strategy. The District continues to collect data to ascertain the level of implementation. Continued monitoring of data will strengthen implementation and assist in making adjustments throughout implementation. Providing additional training as needed will ensure consistency. The District may choose to utilize interventionists to assist in training as strategies are implemented. Setting and communicating expectations, including a time line for full implementation that includes a monitoring process will assist in ascertaining implementation of the strategies. Conversations about expectations for implementation 28

29 may include all instructional staff. Components in Literacy by Design are designed to support instruction of the Five Components (LBD T74). Recommendations for reading block. Conversations revealed that a 90-minute reading block will be implemented in all K-5 classrooms in September Based on input from data collected during this evaluation, the readiness for full implementation was moderate to high. Implementation may be strengthened by communicating clear expectations regarding the number of minutes, which components are expected, and utilization of the same pacing plan for all schools and all teachers. As reading block is implemented, monitoring for consistency and evidence of components will ensure implementation at high levels. The District s approved text resource, Literacy by Design (2008, Harcourt Achieve, Inc.) Program Overview recommends a pacing plan for reading block that supports a balanced literacy approach (p. T36). Reading components include explicit, whole-group instruction with components of modeled reading, shared reading, interactive reading, and independent reading. The overall pacing plan outlines reading instruction coupled with writing instruction that includes explicit instruction in writing skills through modeled writing, shared writing, interactive writing, as well as small-group writing and independent writing (LBD, 2008, p. T-60 T61). Given the goal of the District is to implement a 90-minute block reading program at the elementary schools in , a pacing plan will provide guidance to classroom teachers. LBD suggests a plan that includes a writing program. See Figure 5 for a modified, recommended comprehensive model of literacy instruction and intervention, as suggested by the text series, Literacy by Design. 29

30 Reading Instruction 90 Minutes Time Allotted Skills (Examples) Grouping Build Reading Skills 20 minutes Phonics, Vocabulary, Spelling Read and Comprehend 20 minutes Modeled Reading Target reading skills Differentiated Reading 45 minutes Comprehension, Vocabulary, Phonics, Fluency Support Reading Independence 5 minutes Review strategies learned during the lesson Whole-group Whole-group Small-group Small-group Independent Whole-group Writing + 30 Minutes Time Allotted Skills (Examples) Grouping Build Writing Skills 20 minutes Writing process, skills, and form Whole-group Small-group Write 10 minutes Apply Small-group Independent RtI + 30 minutes 30 minutes Explicit Reading Tier 1, Tier 2, Tier 3 Instruction Intervention Total 150 minutes 150 minutes Instruction & Intervention Flexible Groups Figure 5. Recommended components and time allotment of a literacy block from approved literacy resource, Literacy by Design (p. T44) and includes time for RtI. Reading and writing have a reciprocal relationship, and connecting these two critical aspects of literacy instructionally benefits students. Readers who read like writers have a deeper understanding of text. Writers who write like readers are better able to express themselves (LBD, 2008, p. T12). If the District decides to include a writing program for a 120-minute block, there is a suggested LBD pacing plan. Also, since the District prioritized RtI, it is included in the model above to depict a full 150 minutes of literacy instruction during a daily schedule. Push-in model during reading block. A push-in model was identified as an approach that will strengthen the overall reading program. Ideas to consider prior to implementation might include: 1) Observation of successful programs that will provide models of effective implementation; 2) Continuing conversations about role clarification (that of classroom teacher and reading teacher) and how the push-in model will be 30

31 implemented; 3) An implementation plan to include expectations and how the reading teacher and the classroom teacher will reciprocally teach as a team or as their roles are defined. Recommendation regarding Literacy by Design. Given the District adopted Harcourt s Literacy by Design (2008) as the approved text resource for literacy, all resources were in place in elementary classrooms. A review of the teaching resources revealed a comprehensive literacy program inclusive of English Language Arts skills, essential reading skills as well as a reading and writing plan, with reinforcement and instruction in flexible groups (whole-group, small-group, and independent). In addition, the components of a comprehensive balanced literacy program were in place to ensure reading supports writing and writing supports reading. The prowess of the designers and authors of LBD included such literacy experts as Lynda Hoyt (literacy as a whole), Sharon Hill (writing), Michael Opitz (literacy skills and assessment), Robert Marzano (instruction and vocabulary) and for ELL strategies, David and Yvonne Freeman. It was noted during interviews there were varying thoughts about weak and strong areas of the series. In that case, the District might choose a supplemental series to address the weak areas of the series and to scaffold classroom instruction. If a resource is needed to jump-start a conversation about a balanced literacy program, the LBD Comprehensive Teacher s Guide Professional Handbook is already available in all elementary classrooms. Engaging staff in a collaborative process may clarify the strengths and weaknesses of the Literacy by Design series. Other resources. There were several supplemental resources used throughout the District. Specific to this reading program evaluation, a few people (specifically reading teachers) used Pathways. Harcourt s Leveled Literacy Intervention (LLI), a series authored by Fountas and Pinnell, was just purchased with District grant funds for reading teachers to supplement instruction with students in the special reading class. That decision was made using a collaborative approach that will strengthen implementation. All of these supplements were grounded in reading theory and 31

32 provided research-based strategies. A model of collaboratively choosing the resources may assist in consistent implementation across the District. Summary statement. The intent of this Utilization-Focused Program Evaluation was to inform and confirm processes and practices in the Kearney R-I School District Title I Reading Program. The evaluation was a gap analysis between the current situation of reading instruction, intervention, and implementation of strategies and the desired state of full implementation of specific strategies and initiatives that will catapult young readers toward academic success. Observation results and interviews conveyed a learning culture that engages all students from bell to bell and revealed that educators throughout the District care deeply about each individual student who walks through the halls of the Kearney R-I School District. 32

33 Reference List Buffam, A., Mattos, M., and Weber, C. (2010). The why behind RtI. Interventions that Work, 68(2), p Harcourt. (2008). Literacy by design: Comprehensive teacher s guide: Appendix. China: Harcourt Achieve Inc. Harcourt. (2008). Literacy by design: Comprehensive teacher s guide: Professional handbook. China: Harcourt Achieve Inc. Harcourt. (2008). Literacy by design: Comprehensive teacher s guide: Program Overview Grade 2. China: Harcourt Achieve Inc. International Reading Association. ( ). RtI: A practical manual. PDF: Retrieved March 20, 2014 from: pdf Kearney R-I School District( ). Comprehensive school improvement plan. Retrieved March 7, 2014 from the Kearney R-I School District website: Patton, M. Q. (1997). Utilization-focused evaluation: The new century text. (3rd Ed.) Thousand Oaks, CA: Sage Publications. PBWAS Positive Behavior Intervention and Supports. (2014). Retrieved from PBWAS website on April 29, 2014: Preskill, H. and Torres. R. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage Publications, Inc. 33

34 RC 2 Solutions, LLC. (2013). Kearney R-I School District Curriculum and Assessment Department multi-year data report. Kearney, MO: RC 2 Solutions, LLC. Reeves, D. B. (2004). Assessing educational leaders: Evaluating performance for improved individual and organizational results. Thousand Oaks, CA: Corwin Press. Sparks, D. (2005). Leading for results: Transforming teaching, learning, and relationships in schools. Thousand Oaks, CA: Corwin Press. 34

35 Appendix A Kearney R-I School District Title I Reading Evaluation Plan of Work 35

36 Program Evaluation Kearney R-1 School District: Plan of Work Elementary Reading (Title I) Administrative Contact Project Manager Jacque Banks, Assistant Superintendent of Special Programs Melody A. Smith, EdD, Partner Patron Insight, Inc. Project Program Evaluation for Elementary Reading (Title I) Dogwood Hawthorne Kearney Southview Project Goals 1) Evaluate effectiveness of reading program by considering Structure (i.e., classroom, RTI, push-in/pull-out, flexible groups, tutoring) Reading block Implementation of Learning by Design curriculum/resources Impact of instructional practices (1st year interventionists) Integrity of instructional practice Analysis of data resources (determined by District) Student impact 2) Present findings 3) Recommendations for consideration Timeline February 28, 2014 Administrative consultation March 2014 Collecting, gathering, observing April 2014 On-site wrap up and writing May 2014 Reporting (TBD) 36