Copyright 2011. All rights reserved. 1183_09/11
Contents Page Executive Summary... 1 Implementation Appraisal... 1 Analysis of Key Stakeholder Interviews... 3 Analysis of the RtI Center District Personnel Survey... 4 Analysis of the Foundational Overview Workshop Evaluation Surveys... 6 Analysis of Interviews on the Foundational Overview Workshops... 6 Analysis of Evaluation Surveys from PBIS Trainings... 7 Introduction... 9 Background... 9 Overview of the Year 2 Evaluation... 11 Organization of the Report... 12 Implementation Appraisal... 13 Goal 1: Collaboratively Develop, Implement, and Evaluate Training... 13 Goal 2: Develop and Implement Universal Processes... 16 Goal 3: Develop and Implement a Statewide RtI Technical Assistance Plan... 17 Analysis of Key Stakeholder Interviews... 20 RtI Center Oversight and Support... 20 RtI Center Goals... 21 RtI Center Successes... 22 RtI Center Challenges... 23 Future Work... 25 Analysis of the RtI Center District Personnel Survey... 27 Data Collection... 27 Survey Respondents... 27 Survey Results... 28 Analysis of the Foundational Overview Workshop Evaluation Surveys... 35 Survey Respondents... 35 Survey Results... 37 Copyright 2011. All rights reserved. 1183_09/11
Analysis of Interviews on the Foundational Overview Workshops... 41 Respondents Positions and Roles With Respect to RtI... 41 Level of RtI Implementation... 42 Usefulness of the Foundational Overview Workshop... 42 Usefulness of the Schoolwide Implementation Review Tool... 43 Identification of Priorities and Next Steps for RtI Implementation... 44 Suggestions for Improving the Foundational Overview Workshop... 44 Appendix. Analysis of Evaluation Surveys from PBIS Trainings... 45 August 2010 Coaches Conference... 46 August 2010 Training of Trainers... 49 November 2010 District Leadership Summit... 50 December 2010 Tier 2/Tier 3 Administrative Overview... 52 January 2011 Tier 2/Tier 3 Administrative Overview... 53 March 2011 Tier 3 Training... 55 April 2011 District Leadership Summit... 56 April 2011 External Coaches Forum... 57
Executive Summary Cooperative Educational Service Agency (CESA) 5 contracted with the American Institutes for Research (AIR) to conduct a formative evaluation of Year 2 of the Wisconsin Response to Intervention Center (RtI Center). The purpose of the evaluation was to assess (1) the implementation of the RtI Center s Year 2 goals and activities as documented in its Year 2 work plan; (2) key stakeholders expectations for and knowledge of the RtI Center s work in Year 2; (3) awareness of the RtI Center and Wisconsin s framework for RtI by school district personnel; (4) access to and satisfaction with RtI Center trainings and resources; and (5) the impact of RtI Center trainings on participants (e.g., the extent to which they understand how to assess a school s level of RtI implementation and how to prioritize and plan for RtI implementation). The findings from each component of the evaluation are summarized in this report. Implementation Appraisal The evaluation team conducted an implementation appraisal to determine what progress the RtI Center made in achieving its Year 2 goals. With minor exceptions, the RtI Center was successful in achieving its goals. A summary of the center s progress in completing activities related to each of its major Year 2 goals is summarized in the following subsections. Goal 1: Collaboratively Develop, Implement, and Evaluate Training A primary objective in Year 2 was to develop or adapt content for RtI workshops and trainings either internally or by contracting with representatives of CESAs or other organizations. All Goal 1 activities were completed or nearing completion at the end of Year 2. Staff from the RtI Center worked with the Wisconsin Department of Public Instruction (DPI) and representatives from other statewide initiatives to develop materials for the following workshops, trainings, or online training modules: the Foundational Overview Workshop, the RtI Framework Mapping Workshop, the Professional Learning Communities Training of Trainers Series, four assessment literacy modules, and a family engagement module. The RtI Center also contracted with CESAs and other organizations to develop additional content for trainings or online modules that will be delivered in 2011 12. The following workshops or trainings focusing on academic RtI were offered by RtI Center staff or contractors in 2010 11: a four-part Professional Learning Communities Training of Trainers Series offered in fall 2010 and winter 2011 that was attended by 26 trainers from across the state and 14 Foundational Overview Workshops, offered regionally in spring 2011 through the CESAs, that included more than 1,100 participants from 161 districts throughout the state. The RtI Center also piloted its RtI Framework Mapping Workshop in spring 2011. Several templates (e.g., a project proposal form, a guidance document template, and a contract template) that were created at the end of Year 1 to facilitate work with CESAs were reviewed by the CESA Statewide Network, revised, and used in contracting with various groups for the development of training content in 2010 11. Wisconsin RtI Center: Year 2 Formative Evaluation 1
In spring 2011, RtI Center staff released a draft of the recommended scope and sequence of RtI trainings that will be offered by the RtI Center or statewide trainers. RtI Center staff held meetings with several schools and CESAs throughout the year to obtain feedback on training materials and gather feedback on RtI technical assistance needs. Goal 2: Develop and Implement Universal Processes RtI Center staff continued to work with CESAs in Year 2 to develop a standard set of processes for the development and delivery of RtI trainings. While documentation for some of the processes was completed, additional processes are still in development. According to the director of the RtI Center, this work was delayed because individual commitments had to be negotiated with each CESA. To facilitate this process, a solution-focused group was established that includes a representative from each CESA who can provide RtI Center staff with information on how services are delivered within specific regions of the state. The goal of the group is to review proposed processes and identify solutions for implementing these processes in each CESA. Goal 3: Develop and Implement a Statewide RtI Technical Assistance Plan A primary goal in Year 2 was to promote awareness of the RtI Center s training and technical assistance services. Additional Goal 3 objectives included further developing its governance structure and continuing to develop its infrastructure. The RtI Center made substantial progress in achieving its Goal 3 objectives. The following activities related to these objectives were completed in 2010 11: RtI Center staff participated in more than 40 state conferences and network meetings and established partnerships or collaborations with several state initiatives and organizations. The RtI Center website was launched in March 2011. This website, which had received more than 1,400 unique visits as of June 2011, has been populated with several resources for educators, administrators, and parents, including the RtI Center s e-newsletter. An analysis of Year 1 statewide RtI needs assessment surveys was completed in 2010, and the results were disseminated to key stakeholders. In March and June 2010, RtI Center staff met with the Wisconsin RtI Communication Council, the RtI Center s major external stakeholder group. A primary topic of discussion was expanding the membership of the group and changing its name to the RtI State Leadership Board. The board would meet quarterly and coordinate support for the RtI initiative and its statewide implementation. In 2011, RtI Center staff assisted DPI s Internal RtI Workgroup in developing and disseminating the informational brief Wisconsin Response to Intervention: A Guiding Document and successfully collaborated on submitting an application for statewide technical assistance to the National Center on Response to Intervention (NCRTI). Staff members met with and updated DPI and the CESA Statewide Network on the RtI Center s work, progress, goals, and plans. Wisconsin RtI Center: Year 2 Formative Evaluation 2
In Year 2, the RtI Center continued to develop its infrastructure. An RtI research and evaluation coordinator and five regional technical assistance coordinators were hired in spring 2011. The RtI Center also plans to hire a coaching coordinator and a learning disabilities consultant. Analysis of Key Stakeholder Interviews Key stakeholders of the RtI Center, including RtI Center staff, DPI staff, and CESA administrators, were interviewed about their expectations for and knowledge of the RtI Center s Year 2 goals and its progress in achieving those goals. Interview respondents also were asked about their roles and responsibilities with respect to RtI Center oversight and support, their perceptions of the center s successes and challenges, and their expectations for the work to be accomplished in Year 3. RtI Center Oversight and Support DPI staff, CESA administrators, and RtI Center staff play a variety of roles in planning, oversight, and implementation. DPI personnel serve as consultants, advisors, and liaisons to the RtI Center. Several are members of DPI s Internal RtI Workgroup, which is responsible for developing policy and guidance for the implementation of RtI within the state as whole. CESA administrators work closely with DPI to communicate a clear vision for RtI implementation and identify the types of services that CESAs will provide to school districts. CESAs work together through the CESA Statewide Network in partnership with DPI to ensure that RtI Center services are provided equitably to school districts throughout the state. RtI Center staff members are responsible for implementing DPI s vision and framework for RtI. The director works closely with DPI, CESA administrators, and other key stakeholders to coordinate the statewide rollout of RtI and obtain political support and funding for the RtI Center. RtI Center Successes Overall, the respondents indicated that the RtI Center accomplished most of its Year 2 goals. Major successes identified by the respondents included the following: Collaboration among the RtI Center, DPI, and CESAs in planning and implementing the statewide rollout of RtI Launching and populating both the RtI Center and the Wisconsin Positive Behavioral Interventions and Supports (PBIS) Network websites Developing high-quality training materials and successfully implementing the Foundational Overview Workshop and Professional Learning Communities Training of Trainer Series Developing guidance documents and contractual agreements with CESAs and other organizations to deliver trainings Wisconsin RtI Center: Year 2 Formative Evaluation 3
Developing the RtI Schoolwide Implementation Review (SIR) tool to assist schools in assessing their level of RtI implementation Using the expertise of CESA staff in meeting school district needs RtI Center Challenges Several stakeholders mentioned the following challenges: The misperception of RtI as a special education initiative and confusion over how the new specific learning disabilities (SLD) criteria relate to RtI implementation Timely delivery of RtI training and technical assistance Documentation of universal processes and procedures for the development and delivery of training content Other challenges noted by some stakeholders include the following: Launching the RtI Center s academic component Developing evaluation tools to assess RtI implementation and impact Incorporating culturally responsive practices into RtI The respondents indicated, however, that RtI Center staff members are actively working with DPI and the CESAs to address these challenges. Future Work Stakeholders provided the following suggestions for improving RtI Center services, organization, and staffing: Continue to provide training, professional development, and technical assistance. Develop and refine instruments related to fidelity of implementation, universal screening, data management, and evaluation. Continue building the capacity and infrastructure of the RtI Center. Increase coaching capacity. Continue scaling up the RtI Center s academic component. Make the Wisconsin PBIS Network less dependent on the Illinois and National PBIS Networks. Analysis of the RtI Center District Personnel Survey The District Personnel Survey was disseminated in May 2011 to key district staff members throughout the state. The purpose of the survey was to obtain information about the familiarity of district staff with Wisconsin s vision for RtI and their awareness of the vision, mission, and goals of the RtI Center and the types of training and services provided by the RtI Center. Of the 1,936 district staff members who were sent the survey link, 373 completed the survey, for a response Wisconsin RtI Center: Year 2 Formative Evaluation 4
rate of 19 percent. School districts and individual schools from all 12 CESAs were represented among the respondents. Most of the respondents (83 percent) were principals. The key findings from the survey are as follows: The majority of respondents (62 percent) had reviewed the informational brief Wisconsin Response to Intervention: A Guiding Document or listened to the webinar prepared by DPI summarizing the document; a smaller percentage but still a majority (53 percent) reported that other district staff members had reviewed the document or listened to the webinar. The majority of the respondents (62 percent to 85 percent) reported that they were aware of specific aspects of Wisconsin s framework for RtI (e.g., high-quality instruction, balanced assessment, and collaboration) to a moderate or to a great extent. Three fourths of the respondents reported that the information they had received about Wisconsin s definition of and vision for RtI was mostly clear or completely clear. Most of the respondents (87 percent) reported at least some level of familiarity with the RtI Center, although half of the respondents indicated their familiarity with the RtI Center was minimal. Among respondents who had at least some familiarity with the RtI Center, 46 percent were familiar with the vision, mission, and goals of the Wisconsin PBIS Network, while 37 percent were familiar with the vision, mission, and goals of the RtI Center. The respondents reported that they had learned about the RtI Center in a variety of ways. Half indicated that they had learned about the RtI Center through their local CESA; a similar percentage had learned about the RtI Center through state conferences or professional association meetings. Approximately one third (36 percent) had learned about the RtI Center from DPI and one third (33 percent) from a colleague. Among respondents who were familiar with the RtI Center, 52 percent had visited the RtI Center website, and 46 percent had visited the Wisconsin PBIS Network website. One fourth of the respondents who were familiar with the RtI Center had used the Wisconsin RtI SIR tool, and more than half (54 percent) reported that other staff members in their school district had used this tool. Seventy percent of the respondents who were familiar with the RtI Center had attended the Foundational Overview Workshop; 60 percent to 75 percent had attended the PBIS Tier 1 trainings (Parts 1, 2, or 3). In responding to open-ended questions, several respondents requested clarification of the differences between RtI and PBIS. The respondents also identified a variety of tools and information they would like the RtI Center to disseminate to schools and school districts, including progress monitoring tools, archived webinars, and recommendations regarding interventions that can be used in addressing specific student needs. Wisconsin RtI Center: Year 2 Formative Evaluation 5
Analysis of the Foundational Overview Workshop Evaluation Surveys To determine the effectiveness and perceived quality of the Foundational Overview Workshop, workshop facilitators asked participants to complete evaluation surveys. RtI Center staff provided the evaluation team with completed surveys from 8 of the 14 workshops that were offered. The key findings of the surveys are as follows: The respondents were asked to indicate the extent to which they had gained knowledge or familiarity with concepts presented in the workshop. The respondents reported substantial gains in knowledge and familiarity with all concepts that were presented. With the exception of comparing the Wisconsin s vision for RtI to other models, most respondents (74 percent or more) reported that the workshop activities had either deepened their understanding or had motivated them to use the activities or concepts when they returned to their respective schools. Almost all of the respondents (97 percent) reported that they would recommend the workshop to others. Some who said they would not recommend the workshop explained that the information presented was provided in other workshops or trainings. Several suggestions for improvements to the workshop were provided by the respondents, including spending less time on background information, providing examples from schools that have been successful in implementing RtI, and allowing more time for discussion and teamwork. Analysis of Interviews on the Foundational Overview Workshops AIR conducted follow-up interviews with a random subsample of 18 school or district-level staff members who had participated in a Foundational Overview Workshop to obtain more in-depth feedback on the workshops. The interview respondents were asked about the level of RtI implementation in their school district or school, their role in RtI implementation, and their perceptions regarding the usefulness of the Foundational Overview Workshop and the SIR tool. They also were asked to provide suggestions for improving future workshops. RtI Implementation The majority of the interview respondents were school principals and counselors who said that their role was or will be to provide administrative support, resources, and training for teachers who are implementing RtI. Of the 18 respondents, 10 indicated that their school or school district had formally implemented RtI, 5 said they were at the beginning stages of implementation, and 3 indicated that RtI had not yet been implemented. Usefulness of the Foundational Overview Workshop Twelve of the 18 interview respondents reported that the Foundational Overview Workshop was very useful, and several said the workshop provided clarity regarding RtI implementation. The respondents who said the session was only somewhat useful or not at all useful cited repetition of prior knowledge or a lack of concrete examples as reasons. Wisconsin RtI Center: Year 2 Formative Evaluation 6
The majority of the interview respondents (14 of 18) said that the SIR tool was useful in assessing their school or school district s level of RtI implementation. Of the remaining respondents, two did not participate in the workshop session on the SIR tool, and two indicated the tool was not useful. One of these respondents noted that the tool is designed for the elementary level and was not helpful in assessing RtI implementation at the high school level. Fifteen of the 18 respondents said that they were able to identify priorities and next steps at the workshop, but three respondents indicated that they had not gotten far enough into the discussion to identify priorities and next steps. Requests for Additional Help and Suggestions for Improvement The respondents also were asked to identify areas in which they need help from the RtI Center. Effective tools and assessments, models of successful RtI implementation, and continued training were the needs identified by most of the respondents. Suggestions for improving future workshops included examples of best practices and RtI implementation strategies, additional time for discussion and information sharing, and additional information on assessments. Analysis of Evaluation Surveys from PBIS Trainings The evaluation team analyzed the results of feedback surveys completed by participants in a series of eight trainings offered by the Wisconsin PBIS Network during the 2010 11 school year. The trainings included the August 2010 Coaches Conference and Training of Trainers Workshop, the November 2010 and April 2011 District Leadership Summits, the December 2010 and January 2011 Tier 2/Tier 3 trainings, the March 2011 Tier 3 training, and the April 2011 External Coaches Forum. Overall, participant feedback on the PBIS trainings was very positive. Specific highlights from the evaluation surveys are presented in the Appendix. The following are the key findings: At least three fourths of the respondents reported that they were satisfied or very satisfied with the presenters at the August 2010 Coaches Conference. In open-ended comments, the respondents also provided generally positive feedback on the conference sessions. For the August 2010 Training of Trainers workshop, all the respondents reported that they were either very satisfied (83 percent) or satisfied (17 percent) with the training. For the November 2010 and April 2011 District Leadership Summits, the majority of the respondents agreed or strongly agreed with almost all statements regarding the quality and usefulness of the training. However, 31 percent of the respondents disagreed that November 2010 Summit had met their expectations; 25 percent disagreed that adequate time was provided for questions and discussion at the April 2011 Summit. Overall, feedback on the December 2010 and January 2011 Tier 2/Tier 3 trainings was very positive. For the January 2011 training, however, 29 percent of the respondents disagreed or strongly disagreed that the content was organized and easy to follow. For the March 2011 Tier 3 training, almost all the respondents (88 percent or more) agreed or strongly agreed with the statements regarding the quality and usefulness of the training. Wisconsin RtI Center: Year 2 Formative Evaluation 7
For the April 2011 External Coaches Forum, 86 percent or more of respondents agreed or strongly agreed with all the statements regarding the quality and usefulness of the training. Wisconsin RtI Center: Year 2 Formative Evaluation 8
Introduction CESA 5 contracted with AIR to conduct a formative evaluation of Year 2 of the RtI Center. The purpose of the evaluation was to assess (1) the implementation of the RtI Center s Year 2 goals and activities as documented in its Year 2 work plan; (2) key stakeholders expectations for and knowledge of the RtI Center s work in Year 2; (3) awareness of the RtI Center and Wisconsin s framework for RtI by school district personnel; (4) access to and satisfaction with RtI Center trainings and resources; and (5) the impact of RtI Center trainings on participants (e.g., the extent to which they understand how to assess a school s level of RtI implementation and how to prioritize and plan for RtI implementation). Background The RtI Center was established in fall 2009 as a collaborative effort between DPI and the regional CESAs. The center is funded by a discretionary IDEA (Individuals with Disabilities Education Act) grant to the CESA Statewide Network; CESA 5 serves as the center s fiscal agent. The primary purpose of the center is to build statewide capacity for high-quality technical assistance and professional development to school communities, including resources related to RtI, PBIS, culturally responsive education, and family and community partnerships. The RtI Center is organized into two primary divisions: an academic achievement division and a behavioral division. The Wisconsin PBIS Network serves as the behavioral division of the RtI Center. The director of the RtI Center is responsible for coordinating and integrating the work of the two divisions and working with DPI, the CESAs, and external stakeholders to ensure that a consistent vision for RtI is communicated to school districts throughout the state. DPI s Internal RtI Workgroup is responsible for developing a statewide vision and guidance for RtI. The RtI Center is responsible for implementing that vision and providing training and technical assistance to schools and school districts in partnership with the CESAs. The initial year of the RtI Center (2009 10) was a planning year. The primary goals in Year 1 were to: Develop a plan for the development and statewide delivery of RtI technical assistance based on an assessment of the technical assistance needs of schools and CESAs. Create and implement universal processes for the development and delivery of technical assistance services, including processes for the development and refinement of training content, content delivery, evaluation, and data collection and analysis. Build the infrastructure of the RtI Center by hiring key staff. Develop a plan for collaboratively developing and implementing training around best practices of RtI, including a four-year training timeline. Contract with representatives from various state initiatives to develop or adapt content for statewide training. Additional Year 1 objectives included developing a governance structure for the RtI Center; developing an RtI Center visual graphic, logo, and website; disseminating information about the Wisconsin RtI Center: Year 2 Formative Evaluation 9
RtI Center at state conferences and meetings; coordinating with other statewide initiatives; and developing a plan for scaling up RtI across the state. Overall, the RtI Center was successful in achieving its Year 1 goals. Among the RtI Center s Year 1 accomplishments were the following: The administration and analysis of three statewide needs assessment surveys to schoollevel leaders, CESA administrators, and CESA staff to identify needs related to implementing RtI on a statewide basis. The establishment of Wisconsin RtI Communication Council to synchronize messaging about Wisconsin s vision for RtI. The council functions in an advisory capacity and helps establish priorities for the development and delivery of services (e.g., identifying where there are gaps or overlaps in the provision of RtI supports). There is also an advisory group for the Wisconsin PBIS Network that serves a similar function in helping to set priorities for the statewide implementation of PBIS. The development of the RtI Center s visual graphic and logo, the development of the Wisconsin PBIS Network logo, and the establishment of landing pages for the RtI Center and the Wisconsin PBIS Network websites. The dissemination of information about the RtI Center through state conferences and meetings with representatives from statewide initiatives, including state Title I coordinators, coordinators for the Responsive Education for All Children (REACh) initiative, regional service network coordinators, school improvement service coordinators, and school district administrators. The development of three-year work plans for the RtI Center s academic division and the Wisconsin PBIS Network. The development of agreements with CESAs on how services will be delivered across the state and what training model will be used. This work was essential to moving forward with the creation of universal processes for the development and delivery of training content. The hiring of key RtI Center staff, including the director, the PBIS statewide coordinator, the PBIS research and evaluation coordinator, the RtI Center academic coordinator, an administrative assistant, and five regional technical assistance coordinators for the Wisconsin PBIS Network. The development of a four-year training timeline for academic and behavioral RtI that identifies training content, who will provide the trainings, and who will be trained (e.g., CESA consultants and/or school districts). The development of a project proposal form and template for contracting with representatives from various state initiatives to develop content for statewide training and the initiation of contractual agreements with various groups for the development of training content in 2010 11. In Year 2, the RtI Center moved from the planning stage to the development of training content and the initial implementation of trainings. Staff also worked to create awareness of RtI Center Wisconsin RtI Center: Year 2 Formative Evaluation 10
services and resources among school districts throughout the state; continued to develop training and service delivery processes; and worked with DPI, other statewide initiatives, and professional organizations to provide consistent messaging about Wisconsin s vision and framework for RtI. Overview of the Year 2 Evaluation The Year 2 evaluation of the RtI Center focuses primarily on the center s goals and activities related to the statewide implementation of Wisconsin s framework for RtI, focusing on the implementation of academic RtI. Although the Wisconsin PBIS Network serves as the RtI Center s behavioral division, the network has its own funding stream and submits its own annual evaluation report. The Appendix summarizes findings from evaluation surveys completed by participants in several trainings offered by the Wisconsin PBIS Network during the past year. The Year 2 evaluation addresses the following research questions: 1. What progress has the RtI Center made in achieving its Year 2 goals? Where and why is implementation proceeding as anticipated, and where and why is it not? 2. What do key stakeholders know about the RtI Center work and services? To what extent are they satisfied with the progress the RtI Center has made this year in achieving its Year 2 goals? What are their expectations for the RtI Center s work next year? 3. To what extent are school district personnel aware of Wisconsin s vision for RtI and with the training and resources provided by the RtI Center? To what extent are they satisfied with the clarity of messaging about RtI? 4. To what extent do trainings increase participants understanding of key aspects of Wisconsin s vision and framework for RtI? To what extent are participants satisfied with the trainings and find them useful? 5. What is the perceived impact of RtI Center workshops and trainings on participants? For example, based on their participation in the Foundational Overview Workshop, to what extent do they understand how to assess a school s level of implementation of the Wisconsin framework for RtI? To what extent do they understand how to prioritize and plan for implementation of the Wisconsin framework for RtI? To address these research questions, the following evaluation methods were used: Implementation appraisal. The implementation appraisal assessed the extent to which the RtI Center made progress in achieving its Year 2 goals and identified areas in which work has or has not proceeded as anticipated. Relevant documents, including the RtI Center s Year 2 work plan, drafts of training content, and participant lists for workshops and trainings, were reviewed to evaluate the RtI Center s progress in achieving its Year 2 goals. Key stakeholder interviews. Members of the evaluation team interviewed key stakeholders of the RtI Center, including staff from DPI and the CESAs, about their expectations for and knowledge of the RtI Center s Year 2 goals and its progress in achieving these goals. The purpose of the interviews was to contextualize the RtI Center s progress in relationship to stakeholder expectations. Wisconsin RtI Center: Year 2 Formative Evaluation 11
Survey of key district personnel. School and district leaders across the state were asked to complete an online survey that was designed to obtain information about their familiarity with Wisconsin s vision for RtI; their awareness of the vision, mission, and goals of the RtI Center; and the types of training and services provided by the RtI Center. RtI Center staff provided the evaluation team with a list of key district personnel within the state for use in administering the survey. Analysis of feedback surveys completed by participants in RtI Center workshops. The evaluation team analyzed the results of evaluation surveys administered by RtI Center staff to participants in the Foundational Overview Workshop. This workshop was the first in a series of professional development opportunities offered by the RtI Center (or by trainers under contract to the RtI Center) on implementing Wisconsin s framework for RtI. The survey assessed participants understanding of Wisconsin s vision for RtI, their general satisfaction with the workshop, and the extent to which they understand how to assess their school s level of implementation of Wisconsin s framework for RtI and how to prioritize and plan for RtI implementation. Follow-up interviews with a sample of participants in RtI Center workshops. Follow-up interviews were conducted with a stratified random subsample of participants in the Foundational Overview Workshop to obtain more in-depth information on participants perceptions of the quality and usefulness of the training, skills learned, and suggestions for improvement. Analysis of feedback surveys completed by participants in PBIS trainings. The evaluation team also analyzed the results of feedback surveys completed by the participants in a series of eight trainings offered by the Wisconsin PBIS Network during the 2010 11 school year. The surveys assessed participants overall satisfaction with the trainings as well as their assessment of the quality and usefulness of training content and materials. Organization of the Report A review of the RtI Center s progress in meeting its Year 2 goals is presented in the next section. The findings from interviews with key stakeholders are then presented, followed by a summary of the results of the survey of key district personnel. The next sections present the findings from evaluation surveys completed by participants in the Foundational Overview Workshop and the findings from interviews with a subsample of participants in those workshops. Because the primary focus of the report is on the RtI Center s goals and activities related to statewide implementation of academic RtI, the findings from surveys completed by participants in trainings offered by the Wisconsin PBIS Network are presented in the Appendix. Wisconsin RtI Center: Year 2 Formative Evaluation 12
Implementation Appraisal AIR conducted an implementation appraisal to determine what progress the RtI Center had made in achieving its Year 2 goals. The appraisal was based on a review of relevant documents, including the RtI Center s Year 2 work plan, written updates on progress made in achieving Year 2 goals provided by the director of the RtI Center, training materials developed during Year 2, and interviews with RtI Center staff members. In Year 2, the work of the RtI Center was organized around the following goals: Collaboratively develop, implement, and evaluate training for CESA employees and employees from large school district around the best practices of RtI. Collaboratively develop and implement universal processes for an integrated system of RtI technical assistance. Develop and disseminate RtI technical assistance throughout the state. The activities to be completed for each goal were included in the RtI Center s Year 2 work plan. The following subsections summarize the RtI Center s progress in completing these activities. Goal 1: Collaboratively Develop, Implement, and Evaluate Training A primary objective in Year 2 was to develop or adapt content for RtI workshops and trainings either internally or by contracting with representatives from CESAs or other organizations. Additional Goal 1 objectives included developing or finalizing templates for coordinating work with CESAs (e.g., a project proposal form and a contract template for the development of training content); meeting with representatives of CESAs and school districts to obtain information on the next steps in needed services or training; and creating a training timeline with specific content for the next four years. All Goal 1 activities were completed or nearing completion at the end of Year 2. Each activity is described in greater detail in the following subsections. Developing or Adapting Training Content The RtI Center was successful in contracting with representatives from CESAs or other statewide initiatives in developing or adapting training content in Year 2. Staff from the RtI Center also worked with DPI to develop content for the Foundational Overview Workshop and with REACh coordinators and DPI staff to adapt content for the RtI Framework Mapping Workshop. The following materials were developed in Year 2. Foundational Overview Workshop. RtI Center staff worked with DPI to develop content for the Foundational Overview Workshop. This one-day workshop was developed for building-level leadership teams, including principals, a cross-section of staff, and parents. The workshop is the first in a suggested sequence of RtI professional development opportunities offered by the RtI Center and provides a foundation for starting, exploring, or evaluating implementation of an RtI framework at the school level. Training materials for the workshop include the SIR tool, an online tool that was designed for use by school-based teams to assess their current level of implementation of Wisconsin RtI Center: Year 2 Formative Evaluation 13
the Wisconsin framework for RtI. The RtI Center recommends that the SIR tool be completed annually and used to determine priorities for staff training. The RtI Center is currently developing a plan to validate the tool. The content for the Foundational Overview Workshop was peer reviewed prior to being presented to school districts. Between March 2011 and May 2011, 14 workshops were offered to school-based teams by the RtI Center in partnership with CESAs. More than 1,100 participants from 161 school districts attended. Participants in the Foundational Overview Workshop were asked to complete an evaluation survey, which is standard practice for obtaining feedback from participants in workshops and trainings provided by the RtI Center. The workshops will continue to be offered regionally throughout 2011 12. RtI Framework Mapping Workshop. RtI Center staff worked with DPI and REACh coordinators to adapt content for a workshop on RtI framework mapping. This is the second in a suggested sequence of RtI professional development opportunities offered by the RtI Center. The RtI Center recommends that all schools complete this training after completing the Foundational Overview Workshop. The workshop is designed for grade level/content area leadership teams with expertise in curriculum, instruction, and assessment. The purpose of this one-day workshop is to provide school-based teams with a process for mapping a building s current level of RtI implementation in the areas emphasized by Wisconsin s vision for RtI (multilevel systems of support focusing on the essential elements of high-quality instruction, collaboration, and balanced assessment). The goal of the workshop is to help teams identify specific areas of need, determine what current school structures to build on, and determine next steps for implementation. Training content for the workshop was reviewed, finalized, and piloted in spring 2011. The workshop will be delivered regionally throughout the year by the RtI Center, in partnership with CESAs, beginning in 2011 12. Professional Learning Communities Training of Trainers Series. The RtI Center contracted with CESA 4 and Solution Tree to develop and deliver content for four trainings on professional learning communities. The trainings were offered in September 2010, November 2010, and January 2011. This series provided tools and strategies to help facilitate a collaborative culture within schools that focuses on best practices for RtI. Professional learning communities emphasize a shared vision and mission, collective responsibility for student success, and collaboration and shared decision making. A total of 26 participants attended the trainings two state trainers from each of the 12 CESAs and two district trainers from one large school district. Participants in the training series will provide training to school-based teams within their region or school district. Assessment literacy modules. The RtI Center contracted with school improvement services coordinators to develop a series of online training modules and supporting toolkits on balanced assessment. Four modules are being developed: (1) Fundamentals of Assessment; (2) Assessment of Learning; (3) Assessment for Learning; and (4) Assessment as Learning. As of May 2011, the first module had been completed; the other modules were still in development. These online modules will include a balanced assessment chart developed by DPI that explains the different types of assessments that are recommended for use by school districts (formative, benchmark, and summative) and will address knowledge and skills related to universal screeners and progress monitoring. The modules will also include a needs assessment tool that school districts can use to Wisconsin RtI Center: Year 2 Formative Evaluation 14
assess their balanced assessment system. The modules are scheduled for release in fall 2011 and will be available on the RtI Center s website. Family engagement module. The RtI Center contracted with REACh coordinators to adapt content from the REACh initiative for an online family engagement module. Several statewide organizations and initiatives contributed to the development of the module, including the RtI Center, the Wisconsin PBIS Network, the REACh initiative, DPI s Community Learning and Partnerships Team, Wisconsin FACETS (the Family Assistance Center for Education, Training, and Support), WSPEI (the Wisconsin State Parent Educator Initiative), the Professional Development Hub of the Statewide Personnel Development Grant, the state superintendent s Parent Advisory Council, and Milwaukee Public Schools. The online module and supporting toolkits will provide school-based teams and family and community representatives with information on research-based best practices in family-school-community engagement and will link the essential components of an RtI framework at the universal, supplemental, and intensive levels with family engagement activities identified in Epstein s Six Types of Parent Involvement. 1 As of May 2011, a draft module had been developed. The module is scheduled for release in fall 2011 and will be available on the RtI Center s website. Additional training content. The RtI Center has also contracted with CESAs and other organizations to develop content for the following trainings or online modules that will be developed and delivered in 2011 12: (1) a review of universal instructional practices (to be offered as a trainer of trainer series) that will provide recommendations for analyzing critical elements of a school s universal instructional programs and guidance regarding best practices, the common core, alignment to other state academic standards, differentiated instruction, and assessment; (2) adaptation of the REACh initiative progress monitoring training (to be offered as a trainer of trainer series) that will focus on universal screening; progress monitoring; and data analysis at the student, classroom, school, and district levels; and (3) an interactive Web module that will focus on resources and processes for implementing evidence-based practices as well as procedures for assessing current practices and interventions for evidence of effectiveness. The Creation of Templates for Coordinating Work with CESAs At the end of Year 1, RtI Center staff developed drafts of a project proposal form, a guidance document template, and a contract template to be used in contracting with CESAs for the development of training content. In Year 2, these templates were reviewed by the CESA Statewide Network and were revised based on feedback received from network members. These revised templates were used in contracting with various groups to develop training content (e.g., CESA 4 and Solution Tree for the development of the professional learning communities training content; school improvement services coordinators for the development of the literacy assessment modules). RtI Center staff also developed procedures for peer reviewing and piloting training content as well as the trainer of trainers selection process and rubric. 1 See http://www.csos.jhu.edu/p2000/sixtypes.htm. Wisconsin RtI Center: Year 2 Formative Evaluation 15
The Creation of a Training Timeline with Specific Content In spring 2011, RtI Center staff released a draft of the recommended scope and sequence of RtI trainings that will be offered by the RtI Center or statewide trainers. This document describes three types of trainings: purpose-building activities and supports, infrastructure and implementation trainings, and sustainability trainings. Purpose-building activities include the Foundational Overview Workshop, the RtI Framework Mapping Workshop, and the SIR tool. These activities provide a critical overview of the Wisconsin framework for RtI. Typically, schools that have not yet begun implementing an RtI system spend a year in the purpose-building phase. During Years 2 and 3, building leadership teams can move on to infrastructure and implementation trainings. These trainings focus on universal instructional practices; professional learning communities; screening, monitoring, and data analysis; assessment literacy; and evidence-based practices. Schools use the results of the SIR tool to determine which trainings are most needed to build their capacity to implement a high-quality RtI framework. It may take schools several years to fully implement this framework. Sustainability trainings include training in coaching skills and family engagement strategies as well as examples of high-quality implementation from schools within the state. The draft document also includes implementation timelines and training descriptions. 2 Meetings With CESAs and School Districts RtI Center staff held meetings with several schools and CESAs throughout the year to obtain their feedback on the Foundational Overview Workshop, the SIR tool, and the RtI Center s overall technical assistance plan. This peer review process was also used to gather feedback on RtI technical assistance needs. Goal 2: Develop and Implement Universal Processes RtI Center staff continued to work with CESAs in Year 2 to develop a standard set of processes for the development and delivery of RtI trainings, including processes for the development and refinement of training content; content delivery; piloting new and updated trainings; the training of mentors, content coaches, and trainers; website maintenance and Web-based data collection; data analysis; and evaluation (e.g., of content and delivery, coaching effectiveness, and staff). Documentation for the following processes was completed: A project proposal form for integrating existing content/training not owned by DPI (September 2010) A trainer of trainer memorandum of understanding outlining presentation skills and content area endorsements (September 2010) Development and documentation of processes for local education agencies to request services from Wisconsin PBIS Network regional technical assistance coordinators (October 2010) 2 See http://www.wisconsinrticenter.org/assets/files/trainings%20&%20supports%20description%20doc%20for%20th e%20wi%20rti%20ctr.pdf. Wisconsin RtI Center: Year 2 Formative Evaluation 16
Development of a request for applications for a statewide coaching model vision and long-range coaching networking vision (February 2011) Development of a staff evaluation form based on models from several agencies (winter 2011) The documentation of processes for working with CESAs is still in development. According to the director of the RtI Center, this work has been delayed because CESAs operate independently of each other, and individual commitments have to be negotiated with each CESA before agreedon procedures can be documented. Although the RtI Center has been able to obtain written commitments from several CESA administrators, it became apparent that input was needed from CESA staff members who will actually implement the trainings. To address this need, a solutionfocused team was formed that includes a representative from each CESA who can provide RtI Center staff with information on how services are delivered within specific regions of the state. The goal of the group is to review proposed processes and identify solutions for implementing these processes in each CESA. Goal 3: Develop and Implement a Statewide RtI Technical Assistance Plan A primary goal in Year 2 was to promote awareness of the training and technical assistance services of the RtI Center by participating in state conferences and meetings with representatives from various state initiatives and professional associations and by disseminating information about the RtI Center through its website and e-newsletter. Additional Goal 3 objectives included analyzing state and CESA needs assessment surveys that were administered in Year 1 and disseminating the results to stakeholders; further developing the RtI Center s governance structure; working with DPI s Internal RtI Workgroup to develop and refine the RtI Center s goals and work plan; updating DPI and the CESA Statewide Network on the RtI Center s work; and continuing to develop the RtI Center s infrastructure. The RtI Center made substantial progress in achieving its Goal 3 objectives. Activities related to each objective are described in the following subsections. Promoting Awareness of the RtI Center and Developing Partnerships In Year 2, RtI Center staff participated in more than 40 state conferences and network meetings to promote awareness of the RtI Center. The center also established partnerships or collaborations with several state initiatives and organizations, including the Wisconsin CREATE initiative (Culturally Responsive Education for All: Training and Enhancement), the REACh initiative, the Early Childhood Initiative, the Every Child a Graduate initiative, the Regional Service Network, the School Improvement Services Network, the Title I Network, WSPEI, the Social Emotional Foundation for Early Learning state leadership team, the Student Services/Prevention Wellness Team, Student Intervention Monitoring System consultants, Wisconsin FACETS, Parents Plus Inc., DPI s Community Learning and Partnership Team, and DPI content area specialists. The Development of the RtI Center Website and E-Newsletter The RtI Center website was launched in March 2011 and has been populated with several resources for educators, administrators, and parents. The website includes RtI Center vision and Wisconsin RtI Center: Year 2 Formative Evaluation 17