Department of Research & Program Evaluation (DRPE) Office of Accountability Requests for Flexibility Evaluation Approach APPENDIX A 1
How to Use this Guiding Document The purpose of this reference document is to: A. Provide an overview of the FCS Program Evaluation Strategy (PES) B. To highlight how the FCS PES is applied to the evaluation of school-level Requests for Flexibilities (RFFs) C. To provide schools with examples of indicators that will be used to assess program effectiveness (short-term outcomes) and program impact (long-term outcomes) for different types of RFFs by DRPE A. FCS PROGRAM EVALUATION STRATEGY *Leading Indicators: short-term outputs or outcomes that you believe need to be targeted in a program in order to influence the program goals; typically measures that drive or lead to the performance of lag measures. Leading Indicators* Lagging Indicators* *Lagging Indicators: long-term outcomes that are aligned with the program goals; these are the lag measures that you ultimately want to impact as a result of implementing your program or initiative 2
B. Applying the FCS PES to Evaluation of School-level Requests for Flexibility: As you begin to think about how you will monitor and evaluate the impact of your approved RFF waiver, we encourage you to track your process along the three key domains: Domain 1: Implementation To what extent is your program being implemented as intended? How will you know this? Domain 2: Program Effectiveness To what extent is your program achieving its immediate goals? These are the short-term outcomes (i.e., leading indicators) that will give you a sense of whether your RFF waiver is on track to make its intended impact. These will typically be measurable as early as mid-year of implementation. Domain 3: Program Impact What happened as a result of your initiative? These are the long-term outcomes (i.e., lagging indicators) that will give you a sense of whether your RFF waiver achieved its intended purpose. These will typically be measured as early as the end of year 1 of implementation. 3
C. Examples of Indicators Across the Three Domains of the Program Evaluation Strategy Example A: Teacher Professional Development (PD) Evaluation Question #1: To what extent has the professional development been delivered as intended? Purpose: To understand the essential components of the training that are necessary to achieve the outcomes of interest and to identify opportunities for improvement and modification of future trainings Outcomes of interest: Process (Activities/Output) Did the PD sessions occur as intended? Review of implementation timeline Professional Learning Course Proposals Was the intended content delivered at each of Review of objectives, content and methods for Professional Learning Course the PD sessions? each PD session, and timeframe for each PD Proposals; PD session To what extent did the target participants attend the PD sessions? Who contributed to the development and facilitation of the PD sessions? session Teacher attendance data Names, responsibilities, and role requirements for individuals developing/facilitating trainings agendas PD session attendance records Designated school-level staff Submitted to DRPE: Mid-Year End of Year Evaluation Question #2: To what extent does the professional development training achieve its short-term (immediate) goals? Purpose: To understand the extent to which short-term outcomes are being achieved as a result of the PD sessions and to modify subsequent PD sessions based on participant feedback Outcomes of interest: Short-term and Intermediate by (or sent to) Central office? How satisfied are the teachers with the PD? Feedback from teachers Online standard survey #1 To what extent do the teachers perceive that Feedback from teachers Online standard survey #1 the PD meets their needs? To what extent is there a change in teacher s Teacher perceptions Online standard survey #1 knowledge or attitudes as a result of the PD sessions? Survey opened to schools by early fall of school year; Analyzed by DRPE midyear end of year 4
Evaluation Question #3: What happened as a result of implementing these PD sessions? Purpose: To begin to understand the impact of the PD sessions on different stakeholders, particularly students Outcomes of interest: Intermediate and Long-term To what extent is the learning from the PD being transferred to/applied into the classroom? What are the perceptions of the parents and students to the reduced or shortened days? To what extent is there a change in student s knowledge/attitudes/experiences as a result of teacher PD? Teacher perceptions Online standard survey #1 Feedback from parents/students Online standard survey #2 To be determined in consultation with school (examples: standardized test scores, classroom observation tools, new tools developed to assess skills such as critical thinking, etc ) To be determined in consultation with schools at start of year by (or sent to) Central office? Survey opened to schools by early fall of school year; analyzed by DRPE midyear end of year Survey opened to schools by early spring of school year; analyzed by DRPE end of year Spring Semester Example B: PE Credit Waiver Evaluation Question #1: To what extent has the waiver been exercised as intended? Purpose: To understand the essential components of the program that are necessary to achieve the outcomes of interest and to identify opportunities for improvement and modification of future initiatives Outcomes of interest: Process (Activities/Outputs) To what extent was the waiver exercised as intended? How many students benefited from/participated in the program that resulted from the waiver? (where applicable) To what extent was there appropriate faculty available to facilitate the programs resulting from these waivers? Review of implementation timeline and plan Student participation/attendance Implementation plan/schedule Student attendance/participation Status update submitted to DRPE: Mid-Year End of Year Names, responsibilities, and role requirements for individuals conducting programs 5 records Designated school-level staff
Evaluation Question #2: To what extent does exercising the waiver lead to school s achieving their short-term (immediate) goals? Purpose: To understand the extent to which short-term outcomes are being achieved as a result of the program/initiative and to modify subsequent program sessions based on participant feedback Outcomes of interest: Short-term and Intermediate How satisfied are the (impacted) stakeholders (e.g., teachers, partners) with how these waivers are being implemented in their respective schools? Qualitative feedback from designated schoollevel staff and leaders Qualitative interviews with school-level staff Mid-year and end of year Evaluation Question #3: What happened as a result of implementing these waivers? Purpose: To begin to understand the impact of the PD sessions on different stakeholders, particularly students Outcomes of interest: Intermediate and Long-term To what extent is there a change in student s knowledge/attitudes/experiences as a result of exposure to PE Credit waiver? To what extent did students from different demographics benefit from this program? To be determined in consultation with school (examples: standardized test scores, classroom observation tools, new tools developed to assess skills such as critical thinking, etc ) Examine student participation/enrollment/access records from schools To be determined in consultation with schools at start of year Work with school contact to identify appropriate data source Spring Semester Spring Semester To what extent do the parents/students perceive that these new waivers meet their needs? Feedback from parents /students Online standard survey Shared w/ schools in early spring; analyzed by DRPE at end of year 6