Consortium for Educational Research and Evaluation North Carolina North Carolina New Teacher Support Program Final Race to the Top Evaluation Report Authors: Kevin Bastian and Julie Marks Education Policy Initiative at Carolina, The University of North Carolina at Chapel Hill
Table of Contents Executive Summary... 3 Introduction... 6 Data and Analyses... 8 Evaluation Sample... 8 Data Sources... 9 Analysis... 10 Findings... 11 Implementation... 11 To What Extent was the Implemented as Intended?... 11 To What Extent did the Reach its Target Population?... 11 Knowledge, Skills, and Attitudes... 13 How do Teachers Perceive the Impact of Components on their Knowledge, Skills, and Attitudes about Teaching?... 13 How do Teachers Perceive the Impact of Components on Self-Efficacy and Job Satisfaction?... 15 Teacher Effectiveness... 16 To What Extent does the Impact Teacher Effectiveness, as Measured by Teacher Value Added to Student Achievement?... 16 To What Extent does the Impact Teacher Effectiveness, as Measured by the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings?... 18 Retention... 20 To What Extent does the Impact the Retention of Novice Teachers a) in the Same Schools or LEAs, and b) in the State?... 20 Conclusions... 22 Summary of Findings... 22 Implementation... 22 Teacher Knowledge, Skills, and Attitudes... 22 Teacher Effectiveness... 22 Teacher Retention... 22 Limitations... 23 Recommendations and Next Steps... 23 Appendix A. Evaluation Sample... 25 Appendix B. Data Sources... 30 Appendix C. To What Extent was the Implemented as it was Intended?... 42 Consortium for Educational Research and Evaluation North Carolina
Appendix D. To What Extent did the Reach its Target Population?... 44 Appendix E. How do Teachers Perceive the Impact of the Components on their Confidence, Knowledge, and Skills in Teaching?... 52 Appendix F. How do Teachers Perceive the Impact of the on their Self-Efficacy and Job Satisfaction?... 58 Appendix G. To What Extent does the Impact Teacher Effectiveness as Measured by Teacher Value-Added to Student Achievement?... 61 Appendix H. To What Extent does the Impact Teacher Effectiveness as Measured by the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings?... 82 Appendix I. To What Extent does the Impact the Retention of Novice Teachers to the Same School, LEA, and the State?... 90 Consortium for Educational Research and Evaluation North Carolina 2
NORTH CAROLINA NEW TEACHER SUPPORT PROGRAM: FINAL RACE TO THE TOP EVALUATION REPORT Executive Summary Overview The North Carolina New Teacher Support Program () was developed to provide induction supports to beginning teachers in North Carolina s lowest-achieving schools and to meet two high-priority needs identified by the state s Race to the Top (RttT) grant: (1) helping teachers to succeed during their initial years in teaching; and (2) retaining qualified teachers, particularly in high-need schools. The aims to improve the instructional knowledge, skills, attitudes, effectiveness, and retention of participating teachers through the provision of three support components: institutes (multi-day trainings); instructional coaching; and professional development. This report reflects findings from an independent external evaluation of the conducted by the Education Policy Initiative at Carolina as part of the statewide RttT evaluation undertaken by the Consortium for Educational Research and Evaluation North Carolina (CERE NC). Data and Methods This final evaluation report draws upon the following data sources: (1) participation records from each of the components of the ; (2) Perception of Success Inventory for Beginning Teachers (PSI-BT) survey responses; and (3) student demographics and test scores, classroom rosters, teacher evaluation ratings, EVAAS scores, certified salaries, school personnel, and school characteristics files provided by the North Carolina Department of Public Instruction. With these data sources, the Evaluation Team used a comparison group design to examine: levels of participation in components; teachers perceptions of the impact of program components on their confidence, knowledge, and skills for teaching; teachers perceptions of their self-efficacy and job satisfaction; and the impact of the on teacher value-added, teacher evaluation ratings, and teacher retention. Summary of Findings This report reflects evaluation findings for the two full years of program implementation spanning the 2012-13 and 2013-14 school years. The study design and analysis were developed to address evaluation questions across four overarching areas: implementation; teacher knowledge, skills, and attitudes; teacher effectiveness; and teacher retention. Implementation To what extent was the implemented as intended, and to what extent did it reach its target population? Over the course of the four-year grant period, the was developed, staffed, and implemented from the ground up and has grown to scale, serving over 1,100 teachers in 114 schools as of the 2013-14 school year. There were substantial differences in program participation and implementation across the four regions. As the doubled in Consortium for Educational Research and Evaluation North Carolina 3
size in 2013-14, there was a drop in attendance at institutes and professional development sessions and fewer instructional coach visits per teacher. Teacher Knowledge, Skills, and Attitudes How do teachers perceive the impact of components on their confidence, knowledge, skills, and attitudes toward teaching? There was a statistically significant difference in the proportion of evaluation sample respondents who felt the program components had a positive impact on their teaching, compared with similar services provided by their own school. This significant difference also existed between respondents perceptions of program utility and comparison sample perceptions of analogous school-provided services. Teacher Effectiveness To what extent does the impact teacher effectiveness as measured by teacher valueadded to student achievement (EVAAS) and the North Carolina Educator Evaluation System (NCEES) teacher evaluation ratings? Overall, evaluation sample teachers had significantly higher EVAAS estimates than comparison sample teachers in fifth and eighth grade science. When assessing results by cohort, positive and significant EVAAS results were concentrated within Cohort 1 teachers, while Cohort 2 teachers were generally no more or less effective. Regarding teacher evaluation ratings, there were no significant differences between evaluation sample teachers and comparison teachers in overall models. By cohort, Cohort 1 teachers had significantly higher evaluation ratings on four standards in 2013-14. Teacher Retention To what extent does the impact the retention of novice teachers in the same schools or local education agencies (LEAs) and in the state? Overall, evaluation sample teachers were significantly more likely to return to teaching in North Carolina public schools, to the same LEA, and to the same low-performing school. teachers from both cohorts were significantly more likely than comparison sample teachers to return to the same low-performing school. Limitations Two primary limitations should be taken into account when interpreting the findings in this report: (1) the ability to isolate the impact of the is diminished due to the lack of a comparison group of teachers working in schools similar to those served by the in all ways other than participating in the program; and (2) given the time required for program development and scaling up, the length of the evaluation period is not adequate to assess the effectiveness of a consistent program model implemented as intended for first-, second-, and third-year teachers over the course of three years. Consortium for Educational Research and Evaluation North Carolina 4
Recommendations and Next Steps There are three overarching recommendations for the after the close of the RttT grant: 1. The impacts of the on teacher value-added to student achievement and teacher retention support a recommendation to sustain the program beyond the end of the RttT grant; 2. Findings from this evaluation should be used to explore the disparities in program implementation by region, and moving forward, implementation fidelity should be formally monitored; and 3. Strategies should be explored to counter the decline in program participation and effectiveness seen in 2013-14, such as requiring participating schools and Local Education Agencies (districts) to enforce mandatory participant attendance, and/or concentrating resources toward instructional coaching (the most intensive program component). Consortium for Educational Research and Evaluation North Carolina 5
Introduction North Carolina s $400 million, four-year Race to the Top (RttT) grant was built upon a comprehensive plan to strengthen the education workforce, with the goal of having a great teacher in every classroom and a great principal leading every school. With this end in mind, the North Carolina Department of Public Instruction (NCDPI) identified the state s highest-priority needs, including the recruitment and preparation, equitable distribution, professional development, and induction and retention of high-quality teachers. 1 Data from the 2010 Teacher Working Conditions Survey illustrated the need for induction supports prior to RttT, showing that while nearly all new teachers (93%) are assigned a mentor, almost half (47%) do not have time during the day to meet with their mentors, and one in eight indicate that they never received any additional support as new teachers. 1,2 These supports are particularly needed in high-need schools, where novice teachers are concentrated and where the teacher turnover rate is often greatest. To help teachers succeed during their early-career years and persist in the state s highest-need schools, NCDPI partnered with the UNC General Administration (UNC-GA) to develop and implement the North Carolina New Teacher Support Program (). The is a comprehensive induction program for early-career teachers designed to increase competency in goal-setting, backwards planning and assessment, data-driven decision-making, classroom management, and strategies for success in the school and community. The goal of the is to help participating novice teachers acquire the knowledge and skills necessary to increase the quality of their instruction, raise student achievement, and persist in teaching. The drew from the successful induction model utilized by Teach For America to create a three part program comprised of 1) institutes (multi-day training sessions); 2) intensive face-to-face and virtual instructional coaching; and 3) professional development sessions held throughout the academic year. Implementation of these beginning teacher supports is organized through a central office and regional anchor sites located at four UNC system institutions East Carolina University (ECU), the UNC Center for School Leadership Development (UNC-CSLD), UNC Charlotte (UNCC), and UNCG Greensboro (UNCG). As part of the statewide evaluation of RttT conducted by the Consortium for Education Research and Evaluation North Carolina (CERE NC), 3 the Education Policy Initiative at Carolina (EPIC) was tasked with conducting an independent external evaluation of the from the 2011-12 to the 2013-14 school years. Over this time period, EPIC has collected and analyzed data on program implementation and participation, short-term and intermediate outcomes, and summative impact findings to address the following evaluation questions: 1 NC RttT Proposal, 2010 2 Teacher Working Conditions Survey, 2010 3 The Consortium for Educational Research and Evaluation North Carolina (http://cerenc.org) is a partnership of: the SERVE Center, University of North Carolina at Greensboro; the Education Policy Initiative at Carolina, University of North Carolina at Chapel Hill; and the Friday Institute for Educational Innovation, North Carolina State University. Consortium for Educational Research and Evaluation North Carolina 6
Implementation 1. To what extent was the implemented as it was intended? 2. To what extent did the reach its target population? Teacher Knowledge, Skills, and Attitudes 3. How do teachers perceive the impact of components on their confidence, knowledge, and skills in teaching? 4. How do teachers perceive the impact of components on their self-efficacy and job satisfaction? Teacher Effectiveness 5. To what extent does the impact teacher effectiveness as measured by teacher value-added to student achievement? 6. To what extent does the impact teacher effectiveness as measured by the North Carolina Educator Evaluation System (NCEES) teacher evaluation ratings? Teacher Retention 7. To what extent does the impact the retention of novice teachers in the same schools or LEAs and in the state? Consortium for Educational Research and Evaluation North Carolina 7
Data and Analyses Evaluation Sample For the 2012-13 academic year, the inclusion criteria for the evaluation sample was all first- and second-year teachers who began receiving services by December 2012 and who worked in schools that were eligible for and agreed to participate in the. The exclusion criteria for the 2012-13 evaluation sample were as follows: Teachers who began receiving supports in January 2013 or later; Novice teachers receiving supports in non-rttt schools; The small number of third-year teachers served by the program in 2012-13; and Teach for America (TFA) corps members. For the 2013-14 academic year, the Evaluation Team defined the evaluation sample as all first-, second-, and third-year teachers who began receiving supports by December 2013 and who worked in schools that were eligible for and agreed to participate in the. While this sample includes third-year teachers who were fully served by the program in 2013-14, it again excludes teachers who began receiving supports in January 2014 or later, novice teachers who received supports in non-rttt schools, and TFA corps members. In the 2012-13 academic year, the evaluation sample consisted of 344 teachers working in 59 schools and 16 LEAs, with a large majority (72%) of this treatment sample being first-year teachers. In the 2013-14 academic year, the evaluation sample consisted of 808 teachers working in 91 schools and 25 LEAs. Approximately 50% of the treatment sample were first-year teachers, 36% were second-year teachers, and 15% were in their third year of teaching. To best isolate the impact of the, the Evaluation Team used a comparison group design to compare findings between program participants and comparison teachers who did not receive the intervention. An optimal comparison group would be comprised of teachers who were identical to the evaluation sample in every way other than participating in the ; however, identifying such a group poses a particular challenge for evaluations of programs that are universally offered within the eligible target population, leaving two primary options for comparison groups: (1) those who were not eligible for the program but are thought to be similar due to characteristics that place them narrowly outside the program criteria, and (2) those who are eligible for the program but did not participate. To assess the impact of the, the Evaluation Team created two comparison groups. The group included in the main body of this final evaluation report meets the criteria of Option One above schools that were in the bottom decile of school performance in 2011-12 but that were not eligible for and did not receive services. The key limitation of this sample is that it is comprised of schools that did not receive any RttT services, meaning comparisons to this group do not allow for isolation of the impact of the from other RttT programs. Consortium for Educational Research and Evaluation North Carolina 8
In the 2012-13 school year, this comparison group (labeled Non-RttT throughout the report) consisted of all of the first- and second-year teachers who began working in these schools by December 2012. In the 2013-14 school year, this comparison group consisted of all of the first-, second-, and third-year teachers who began working in these schools by December 2013. 4 Overall, the Non-RttT evaluation sample included 1,033 teachers working in 147 schools and 48 LEAs in the 2012-13 school year and 1,507 teachers working in 149 schools and 48 LEAs in the 2013-14 school year. Table 1 presents school characteristics for the evaluation sample and the Non-RttT evaluation sample. Across 2012-13 and 2013-14, teachers worked in schools with more students qualifying for subsidized school meals, more racial and ethnic minority students, a higher short-term suspension rate, and a lower school performance composite than did comparison teachers. In 2012-13, schools also had a significantly higher percentage of novice teachers. Please see Appendix A for more details on the evaluation sample, both comparison samples, 5 and additional characteristics of the treatment and comparison sample schools. Table 1. School Characteristics for the and Non-RttT Group 2012-13 School Year 2013-14 School Year Non-RttT Non-RttT Characteristic Free and Reduced-Price Lunch Percentage 93.58 85.93 ** 91.78 87.40 ** Racial/Ethnic Minority Percentage 90.87 78.37 ** 89.63 78.67 ** Short-Term Suspension Rate (Per 100 students) 48.86 33.52 ** 38.14 25.50 ** Performance Composite 20.04 23.25 ** 31.07 34.29 ** Novice Teacher Percentage 32.99 27.19 ** 32.93 30.29 Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Data Sources The Evaluation Team collected the data presented in this final evaluation report from the UNC- GA implementation team, the evaluation sample teachers and comparison sample teachers, and administrative datasets supplied by NCDPI. Specifically, this final evaluation report draws upon the following data sources: (1) participation records from each of the components of the (institutes, instructional coaching, and professional development); (2) the Perception of Success Inventory for Beginning Teachers (PSI-BT) survey responses of evaluation sample teachers and Non-RttT sample teachers; and (3) student test scores, student demographics, classroom rosters, teacher evaluation ratings, EVAAS scores, certified salaries, school personnel, and school characteristics files provided by NCDPI. 4 Like the evaluation sample, the Non-RttT evaluation sample excludes TFA corps members. 5 The second sample is identified as the Eligible sample. Consortium for Educational Research and Evaluation North Carolina 9
With these data sources, the Evaluation Team examined: levels of participation in components; teachers perceptions of the impact of program components on their confidence in, knowledge of, and skills for teaching; their self-efficacy and job satisfaction; and the impact of the on teacher value-added measures, teacher evaluation ratings, and teacher retention. Please see Appendix B for more details on program documents provided by the UNC-GA implementation team and for PSI-BT survey items, response rates, and respondent characteristics. Analysis This summative report utilizes data collected over a two-year period from two cohorts and across four regional sites. While the was implemented using a structured program model across regions and over time, natural variation is to be expected based upon the different needs and contextual factors within each region. Furthermore, due to the annual scaling-up of the program, different cohorts were exposed to different levels of program intensity. To account for these differences, the Evaluation Team will present impact results for teacher value-added measures, teacher evaluation ratings, and teacher retention in the following groupings: Overall: To determine the overall impact of the, the Evaluation Team will present results for the and Non-RttT evaluation sample teachers using pooled data from the 2012-13 and 2013-14 school-years. By Region: To examine any regional differences in outcomes, the Evaluation Team will present results, separately, for each of the four regions served by the anchor institutions at East Carolina University (ECU), the University of North Carolina Center for School Leadership Development (UNC-CSLD), the University of North Carolina at Charlotte (UNCC), and the University of North Carolina at Greensboro (UNCG). These results use pooled data from 2012-13 and 2013-14. By Cohort: To examine the impact of the based on when a teacher entered the program, the Evaluation Team will present results, separately, for teachers who entered the program in 2012-13 (Cohort 1) and those who entered the program in 2013-14 (Cohort 2). Appendices G through I show value-added, evaluation rating, and retention results in reference to a second comparison group (the Eligible sample), display results for first-year teachers only, and include results from analyses with a more comprehensive set of NC NTSP and comparison sample teachers. Consortium for Educational Research and Evaluation North Carolina 10
Findings Implementation To What Extent was the Implemented as Intended? After the receipt of RttT grant funds, the was developed from the ground up in the 2010-11 school year. As such, the model was scaled up over time as the program increased capacity and there was increased buy-in from Local Education Agencies (LEAs) 6 and schools across the state. As shown in Table 2, there was a 30-fold increase in the number of teachers served by the NC NTSP over the course of the grant period. By 2013-14, the provided induction services to 1,108 teachers in 114 schools and 27 LEAs. Overall, the provided induction services to novice teachers in 73 RttT schools in 2012-13 and 91 RttT schools in 2013-14. 7 By the end of the RttT funding period, this coverage includes over three-quarters of the lowest-achieving schools the program was tasked to serve. Please see Appendix C for more details on the implementation of the. Table 2. Program Implementation Year Teachers Served Schools Served LEAs Served 2010-11* N/A N/A N/A 2011-12 35 13 5 2012-13 542 77 19 2013-14 1108 114 27 * Note: The 2010-11 school year was used for program development, and focused on program planning, staffing, identification of anchor institutes, and school recruitment. To What Extent did the Reach its Target Population? The is comprised of three components: institutes (multi-day trainings); instructional coaching; and professional development. In 2012-13, institutes were held at one centralized location at the University of North Carolina at Chapel Hill; in 2013-14, there were regional institutes held prior to the start of the school year, followed by a centralized institute in September. All instructional coaching interactions and professional development opportunities were organized and implemented at the regional level. 6 LEA is North Carolina s term for traditional school districts and charter schools. 7 There were a small number of schools served by the that were not in the RttT sample. Please see Appendix C for more details. Consortium for Educational Research and Evaluation North Carolina 11
As shown in Table 3, the attendance and reach for all three components notably declined between the 2012-13 and 2013-14 school years. In addition, there was notable variability across regions in program attendance and reach. In the 2012-13 school year, nearly 55% of all evaluation sample participants attended an institute. There was considerable variation between regions, however, with almost 75% of teachers from the ECU region attending an institute, compared with only 40% of teachers in the UNCG region. In 2013-14, institute attendance decreased by over 50% across all regions and fell to 21% overall. Table 3. Evaluation Sample Participation Institute Attended a 2012-13 Institute Attended a Overall 2013-14 Institute Instructional Coaching Average Number of In-Person Coaching Visits Per Month in 2012-13 Average Number of In-Person Coaching Visits Per Month in 2013-14 Professional Development Average Number of PD Sessions Attended in 2012-13 Average Number of PD Sessions Attended in 2013-14 ECU Region UNC-CSLD Region UNCC Region UNCG Region 54.94% 74.24% 66.25% 46.67% 39.74% 20.54% 27.51% 18.99% 16.48% 19.50% 3.73 6.02 2.10 3.39 4.01 2.39 3.82 1.65 2.56 1.77 2.22 2.41 3.59 1.17 2.26 0.76 1.03 1.22 0.14 0.52 The middle panel of Table 3 reflects a decrease in the frequency of instructional coaching visits between the 2012-13 and 2013-14 school years and shows substantial variation in the average number of in-person coaching visits between regions. Program-wide, teachers averaged 1.3 fewer in-person instructional coaching visits per month in 2013-14. 8 Coaching visits declined across all regions, with the largest decreases in the UNCG and ECU regions. In the 2012-13 and 2013-14 school years, the ECU region had the highest average number of instructional coaching visits, while the UNC-CSLD had the lowest average number of instructional coaching visits. 8 The drop in the average number of in-person coaching visits was not due to the serving third-year teachers in 2013-14 (a teacher group that averaged fewer coaching visits overall); rather, the average number of inperson instructional coaching visits declined for first- and second-year teachers. Consortium for Educational Research and Evaluation North Carolina 12
To further illustrate the variation in program intensity across regions, Figure 1 displays the average number of total contact hours (in-person and virtual) between instructional coaches and evaluation sample teachers in the 2013-14 school-year. 9 Overall, teachers in the ECU and UNCC regions averaged 41 and 34 contact hours, respectively, while teachers in the UNC- CSLD and UNCG regions averaged approximately 10 contact hours. Finally, the bottom panel of Table 3 (above) reports the average number of professional development sessions participants attended. Overall, professional development attendance was low, with a marked decline in the 2013-14 school year. In 2012-13, teachers averaged 2.2 professional development sessions (out of six), with a high of 3.59 sessions in the UNC-CSLD region. 10 In 2013-14, teachers averaged 0.76 professional development sessions (out of six), with a high of 1.22 in the UNC-CSLD region and a low of 0.14 in the UNCC region. Please see Appendix D for more detailed information on participation in components. Figure 1. Average Total Contact Hours with Instructional Coaches (2013-14) 45 40 35 30 25 20 15 10 5 0 41.16 33.61 11.35 10.20 ECU Region UNC-CSLD Region UNCC Region UNCG Region Knowledge, Skills, and Attitudes How do Teachers Perceive the Impact of Components on their Knowledge, Skills, and Attitudes about Teaching? A survey assessing teachers perceptions of their knowledge, skills, and attitudes was administered to the evaluation sample and the Non-RttT evaluation sample teachers. The Evaluation Team asked questions to reflect items specific to participants alone (e.g., related to institutes or instructional coaching), as well as items that allowed and comparison sample teachers to respond (e.g., concerning analogous support services provided by their schools or LEAs). Respondents were asked the extent to which they agreed that each component (or analogous school-/lea-provided 9 The UNC-GA program implementers only tracked and reported contact hours for the 2013-14 school year. 10 In 2012-13, several LEAs and schools in the UNC-CSLD region required their teachers to attend the professional development sessions. Consortium for Educational Research and Evaluation North Carolina 13
component) had been helpful in developing their confidence, knowledge, and skills in teaching. Given the potential for response bias, due in part to lower than desirable response rates (see Appendix B), survey results should be interpreted with caution. The top panel of Table 4 presents the summative survey item for institutes. In 2012-13 and 2013-14, approximately 80% of survey respondents agreed or strongly agreed that the institute was helpful in developing their confidence, knowledge, and skills in teaching. Within years, however, there was considerable variation in responses by region, ranging from 59% to 94% in 2012-13 and from 65% to 100% in 2013-14. The middle panels of Table 4 display responses to two items related to instructional coaching. In both the 2012-13 and 2013-14 school years, 77% of the evaluation sample respondents agreed or strongly agreed that the program s instructional coaching was helpful in developing their confidence, knowledge, and skills in teaching. In 2012-13 and 2013-14, approximately 60% of evaluation sample respondents attributed quite a bit or a great deal of their teaching success to help from their instructional coaches. Table 4. Summative Survey Items Overall ECU Region UNC-CSLD Region UNCC Region UNCG Region Institute was helpful in developing my confidence, knowledge, and skills in teaching Percentage Agree or Strongly Agree in 2012-13 80.00 81.58 59.09 93.93 76.47 Percentage Agree or Strongly Agree in 2013-14 82.48 83.72 64.70 75.00 100.00 Instructional Coaching was helpful in developing my confidence, knowledge, and skills in teaching Percentage Agree or Strongly Agree in 2012-13 77.58 70.21 72.72 75.00 94.60 Percentage Agree or Strongly Agree in 2013-14 77.05 81.25 66.21 78.16 72.12 Of the success you have had as a beginning teacher, what amount would you attribute to help from your Instructional Coach Percentage Responding Quite a Bit or a Great Deal 62.43 53.19 45.45 66.67 83.78 in 2012-13 Percentage Responding Quite a Bit or a Great Deal in 2013-14 58.20 65.98 52.71 52.88 55.21 Professional Development was helpful in developing my confidence, knowledge, and skills in teaching Percentage Agree or Strongly Agree in 2012-13 87.12 87.18 80.77 87.50 91.43 Consortium for Educational Research and Evaluation North Carolina 14
As a basis of comparison, Figure 2 displays the responses of evaluation sample teachers to comparable items about their school-provided mentors. While 77% of respondents indicated that their instructional coaches were helpful in developing their confidence, knowledge, and skills in teaching, only 60% of respondents answered similarly about their school-provided mentors. Likewise, while approximately 60% of respondents attributed quite a bit or a great deal of their teaching success to their instructional coaches, less than half of the respondents answered similarly for their school-provided mentors. Both of these differences were statistically significant. The last panel of Table 4 (above) presents the summative survey item for professional development. In 2012-13, 87% of evaluation sample respondents agreed or strongly agreed that the program s professional development sessions were helpful in developing their confidence, knowledge, and skills in teaching. 11 Please see Appendix E for more details on teachers perceptions of program utility. Figure 2. Teacher Perceptions of Quality Relative to School-Provided Support Agrees/Strongly Agrees that Coach/Mentor Helpful in Developing Confidence, Knowledge, and Skills in Teaching Attributes "Quite a Bit" or "A Great Deal" of Success to Coach/Mentor 60.38 60.10 62.43 58.10 45.92 49.09 77.58 77.05 0 10 20 30 40 50 60 70 80 90 -Provided 2012-13 -Provided 2013-14 School-Provided 2012-13 School-Provided 2013-14 How do Teachers Perceive the Impact of Components on Self-Efficacy and Job Satisfaction? The Evaluation Team assessed teachers perceptions of self-efficacy and job satisfaction with five previously-validated items on the PSI-BT. As shown in Table 5 (following page), in both 2012-13 and 2013-14, the percentages of evaluation sample respondents who agreed or strongly agreed with the self-efficacy and job satisfaction items were higher than the percentages of Non-RttT evaluation sample respondents who did so. In 2012-13, NC 11 Due to an error in the response categories recorded for the 2013-14 summative professional development question, the Evaluation Team is unable to present 2013-14 results. Consortium for Educational Research and Evaluation North Carolina 15
NTSP evaluation sample respondents registered significantly higher levels of agreement (84% to 78%) with the self-efficacy item, I feel inspired to instruct students to the best of my ability. In 2013-14, evaluation sample survey respondents rated their self-efficacy and job satisfaction significantly higher than did comparison sample respondents on all five items. These differences should be interpreted carefully, however, due to the low response rate for Non-RttT teachers and the potential for bias in the sample of teachers who responded. See Appendix B for more details on these response rates and differences between teachers who did and did not respond to the PSI-BT, and see Appendix F for more details on the and Non-RttT evaluation sample responses to the self-efficacy and job satisfaction items. Table 5. Self-Efficacy and Job Satisfaction Survey Items Survey Items I am able to successfully teach students with a variety of ability levels I am able to motivate all students I feel inspired to instruct students to the best of my ability In general, I am satisfied with my current job I consider teaching to be my ideal career 2012-13 Percentage Agree or Strongly Agree 2012-13 2013-14 2013-14 78.92% 72.64% 84.07% 74.16% 60.24% 54.07% 68.87% 54.50% 84.15% 77.92% 81.52% 75.14% 59.15% 54.55% 56.76% 47.75% 70.73% 67.20% 68.07% 60.68% Teacher Effectiveness To What Extent does the Impact Teacher Effectiveness, as Measured by Teacher Value Added to Student Achievement? To assess the contributions of teachers to student achievement, the Evaluation Team analyzed EVAAS teacher effectiveness estimates. The Evaluation Team combined EVAAS data from elementary and middle grades to run separate models for mathematics, reading, and science (End-of-Grade [EOG] science exams in grades five and eight), and combined EVAAS data from End-of-Course (EOC) exams (Biology, English II, and Math I) to run a single EOC model. For these models, the Evaluation Team made teacher EVAAS estimates the outcome variable and regressed this measure of value added to student achievement on a set of school characteristics and teacher experience. The Evaluation Team chose to control for these variables due to the significant differences in school characteristics between and comparison sample schools, and because these contextual variables are not accounted for in EVAAS models. Results from these models express the adjusted-average differences in student achievement in normal Consortium for Educational Research and Evaluation North Carolina 16
curve equivalency units between students taught by teachers and students taught by Non-RttT sample teachers. The top portion of Table 6 displays overall EVAAS results for evaluation sample teachers in comparison to Non-RttT evaluation sample teachers. Over the 2012-13 and 2013-14 school years, students taught by evaluation sample teachers in fifth and eighth grade science made significantly larger achievement gains than did students taught by Non-RttT evaluation sample teachers. There were no significant differences for EOG mathematics, EOG reading, and all EOC exams. The bottom portion of Table 6 illustrates the heterogeneity of value-added results across regions. Students taught by evaluation sample teachers in the ECU region made significantly larger achievement gains in mathematics, reading, and fifth and eighth grade science than did students taught by Non-RttT evaluation sample teachers. Conversely, students taught by evaluation sample teachers in the ECU region made significantly smaller achievement gains on EOC exams. evaluation sample teachers in the UNCC region were significantly more effective in elementary and middle grades mathematics, and evaluation sample teachers in the UNCG region were significantly more effective in courses with EOC exams. Table 6. Overall and Regional EVAAS Results (2012-13 and 2013-14) vs. Non- RttT Group (2012-13 and 2013-14) Elementary and Middle Grades Mathematics Elementary and Middle Grades Reading 5 th and 8 th Grade Science EOC Exams Overall 0.828 0.259 1.011 + 0.083 ECU Region 2.219 ** 1.116 ** 2.772 * -0.652 * UNC-CSLD Region -0.007-0.161 1.052-0.727 UNCC Region 1.915 + 0.236 0.234-0.370 UNCG Region -1.033-0.336-0.051 1.214 * Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 17
Table 7 presents considerable variability in teacher value added to student achievement by NC NTSP cohort. Cohort 1 teachers first served by the program in 2012-13 were significantly more effective than Non-RttT evaluation sample teachers in elementary and middle grades mathematics and reading in 2012-13 and when pooling data for two years (2012-13 and 2013-14). Teachers from both cohorts were more effective in fifth and eighth grade science in 2013-14. There were no other significant differences for Cohort 2 teachers. Table 7. EVAAS Results by Cohort Elementary and Middle Grades Math Elementary and Middle Grades Reading 2012-13 2012-13 Cohort 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 Cohort 1 3.054 ** 0.639 1.483 + 1.670 ** 0.307 0.759 * Cohort 2 --- -0.189-0.011 --- -0.183-0.273 5 th and 8 th Grade Science EOC Exams Cohort 1 0.971 1.208 + 1.127 0.693-0.088 0.029 Cohort 2 --- 1.261 + 0.862 --- -0.122 0.202 Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Please see Appendix G for: (1) more details on the value-added methodology; (2) a complete set of EVAAS results, including both comparison groups and models focusing on first-year teachers only; (3) an alternative set of value-added models with student-level data; and (4) value-added results with a more comprehensive set of and comparison sample teachers. To What Extent does the Impact Teacher Effectiveness, as Measured by the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? Since many important aspects of teaching will not be fully captured by measures of teachers value added to student achievement, the Evaluation Team analyzed teachers evaluation ratings on the five North Carolina Professional Teaching Standards (NCPTS), all of which are directly assessed by school administrators. For these analyses, the Evaluation Team estimated models for which the outcome variable was a teacher s evaluation rating on a one-to-five scale (where one was Not Demonstrated and five was Distinguished) and controlled for teacher experience and school characteristics. Results from these models estimate the odds of evaluation sample teachers receiving higher evaluation ratings than their Non-RttT evaluation sample peers. Statistically significant results greater than one indicate higher evaluation ratings; statistically significant results less than one indicate lower evaluation ratings. Consortium for Educational Research and Evaluation North Carolina 18
Over the 2012-13 and 2013-14 school years, the top panel of Table 8 shows no significant evaluation rating differences between evaluation sample teachers and Non-RttT evaluation sample teachers. When analyzing the data by region, the bottom panel of Table 8 shows that teachers in the UNCC region had significantly higher evaluation ratings on Leadership (Standard 1) and Reflecting on Practice (Standard 5). Table 8. Overall and Regional Evaluation Rating Results (2012-13 and 2013-14) vs Non-RttT (2012-13 and 2013-14) Standard Standard Standard Standard Standard 1 2 3 4 5 Overall 1.072 0.960 0.993 0.975 0.996 ECU Region 0.878 0.921 0.666 0.655 0.941 UNC-CSLD Region 0.834 0.719 0.906 0.896 0.886 UNCC Region 1.822 + 1.601 1.613 1.588 1.735 + UNCG Region 0.928 0.771 0.931 0.913 0.675 Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. To determine whether evaluation ratings differ by cohort, Table 9 presents separate evaluation rating results for Cohorts 1 and 2. For Cohort 1, there were no significant differences in 2012-13; however, in their second year in the program (2013-14), Cohort 1 teachers had significantly higher evaluation ratings on Leadership (Standard 1), Content Knowledge (Standard 3), Facilitating Student Learning (Standard 4), and Reflecting on Practice (Standard 5). In contrast, Cohort 2 teachers had significantly lower evaluation ratings for Classroom Environment (Standard 2) in 2013-14 and when pooling data for two years (2012-13 and 2013-14). Please see Appendix H for: (1) more details on the evaluation rating models; (2) a complete set of evaluation rating results, including both comparison groups and models focusing on first-year teachers only; and (3) evaluation rating results with a more comprehensive set of and comparison sample teachers. Table 9. Evaluation Rating Results by Cohort Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 Cohort 2012-13 School Year Cohort 1 1.082 1.128 0.997 0.842 0.980 Cohort 2 --- --- --- --- --- 2013-14 School Year Cohort 1 1.722 * 1.388 1.709 + 1.844 * 1.572 + Cohort 2 0.845 0.700 + 0.748 0.811 0.801 2012-13 and 2013-14 School Years Cohort 1 1.376 1.267 1.323 1.203 1.257 Cohort 2 0.779 0.658 * 0.694 0.752 0.730 Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 19
Retention To What Extent does the Impact the Retention of Novice Teachers a) in the Same Schools or LEAs, and b) in the State? To determine whether evaluation sample teachers were more likely to remain in teaching than their Non-RttT evaluation sample peers, the Evaluation Team estimated models for three types of retention: (1) returning to any North Carolina public school in the following school year (2013-14 and 2014-15); (2) returning to the same LEA in the following school year; and (3) returning to the same low-performing school in the following school year. For these analyses, the outcome variable was a 1 if the teacher returned in the following school year and a 0 if the teacher did not return. In these models, the Evaluation Team controlled for teacher experience and school characteristics and, post-estimation, converted the results to predicted retention probabilities to facilitate easier interpretation of the results. Overall, when pooling data from the 2012-13 and 2013-14 school years, evaluation sample teachers were significantly more likely to return to a North Carolina public school and to their same LEA than were Non-RttT evaluation sample teachers. However, given the need to keep teachers in high-need environments, the most important retention results are for teachers returning to the same low-performing school. As shown in Figure 3 (following page), the program-wide predicted probability of school-level retention was 73% significantly higher than for the Non-RttT evaluation sample (66%). These school-level predicted probabilities of retention were above 70% in all four regions and were significantly higher than those for Non-RttT evaluation sample teachers in the ECU, UNCC, and UNCG regions. Examining school-level retention by cohort, Figure 4 (following page) indicates that Cohort 1 teachers had significantly higher retention rates than did Non-RttT evaluation sample teachers. When school-level retention was calculated for Cohort 2 in the 2014-15 school year, results demonstrated that they also had significantly higher retention rates than did Non-RttT evaluation sample teachers. Please see Appendix I for: (1) more details on the retention models; (2) a complete set of retention results, including both comparison groups and models focusing on first-year teachers only; and (3) retention results with a more comprehensive set of and comparison sample teachers. Consortium for Educational Research and Evaluation North Carolina 20
Predicted Probability of Returning Predicted Probability of Returning : Final RttT Evaluation Report Figure 3. School-Level Retention (2013-14 and 2014-15) 80 73.04** 77.26** 70.22 72.66+ 72.35+ 65.78 60 40 20 0 Overall ECU Region UNC-CSLD Region UNCC Region UNCG Region Non-RttT Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Figure 4. School-Level Retention Results by Cohort 80 74.11+ 67.14 75.76** 74.18** 71.12* 71.67* 64.92 65.78 60 40 20 0 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Cohort 1 Cohort 2 Non-RttT Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Cohort 2 does not have a bar for returning in 2013-14 because they entered the program in 2013-14. Consortium for Educational Research and Evaluation North Carolina 21
Conclusions Summary of Findings Implementation Over the course of the four-year grant period, the was developed, staffed, and implemented from the ground up and has grown to scale, serving over 1,100 teachers in 114 schools as of the 2013-14 school year. There were substantial differences in program participation and implementation across the four regions. Furthermore, as the NC NTSP doubled in size in 2013-14, there was a drop in attendance at institutes and professional development sessions, as well as fewer instructional coach visits per teacher. Teacher Knowledge, Skills, and Attitudes A large majority of evaluation sample teachers who responded to the PSI-BT survey felt that the program components helped their confidence, knowledge, and skills in teaching. In addition, there was a statistically significant difference in the proportion of evaluation sample respondents who felt the program components had a positive impact on their teaching, compared to similar services provided by their own school. This significant difference also existed between perceptions of program utility and Non-RttT sample perceptions of analogous school-provided services. Teacher Effectiveness Overall, evaluation sample teachers were significantly more effective than Non-RttT evaluation sample teachers in fifth and eighth grade science, based on students End-of-Grade exam results. When assessing results by cohort, significant EVAAS results were concentrated within Cohort 1 teachers Cohort 1 teachers were significantly more effective in elementary and middle grades mathematics and reading (both in 2012-13 in isolation and with two years of pooled data) and were more effective in fifth and eighth grade science in 2013-14. There was only one positive and significant result for Cohort 2 teachers, who were significantly more effective than Non-RttT evaluation sample teachers in fifth and eighth grade science in 2013-14. In terms of teacher evaluation ratings, there were no significant differences between evaluation sample teachers and Non-RttT evaluation sample teachers in the overall models. By cohort, Cohort 1 teachers had significantly higher evaluation ratings on four North Carolina Professional Teaching Standards (Leadership, Content Knowledge, Facilitating Student Learning, and Reflecting on Practice) in the 2013-14 school year, while Cohort 2 teachers had significantly lower evaluation ratings on a fifth Standard (Classroom Environment). Teacher Retention Overall, evaluation sample teachers were significantly more likely to return to teaching in North Carolina public schools, to the same LEA, and to the same low-performing Consortium for Educational Research and Evaluation North Carolina 22
schools in comparison to Non-RttT evaluation sample teachers. Focusing on schoollevel retention, teachers from both cohorts were significantly more likely to return to the same low-performing school than were Non-RttT evaluation sample teachers. Limitations Two primary limitations should be taken into account when interpreting the findings in this report: 1. The Evaluation Team s ability to isolate the impact of the is diminished by two factors. First, the serves schools that received the highest proportion of RttT interventions, some of which District and School Transformation are providing support services similar to those provided by the. Thus, choosing comparison sample schools outside of this eligible sample, like the Non-RttT group, precludes the ability to hold these additional RttT interventions constant. Second, the is universally offered in all of North Carolina s lowest-achieving schools, meaning comparisons with schools that are eligible yet non-participating may not take into account potential confounding factors related to selection into the program. If there are differences between the sample and the comparison groups that the Evaluation Team does not control for and that influence the outcomes of interest, then the evaluation results will be biased. 2. A second challenge of this evaluation lies in the fact that the data used in these analyses represent three program components (institutes, instructional coaching, and professional development) implemented across four regional sites with two cohorts of teachers and in a model that has greatly expanded in scope over the course of the evaluation period. Simply put, given the time required for program development and scaling up, a longer evaluation period is needed to assess adequately the effectiveness of a consistent program model implemented as intended for first-, second-, and third-year teachers over the course of three years. Recommendations and Next Steps There are three overarching recommendations for the after the close of the RttT grant: 1. Given the overwhelming need for effective teachers who remain in low-performing schools, the impacts of the on teacher value added to student achievement and teacher retention support a recommendation to sustain the program beyond the end of the RttT grant. 2. Findings from this evaluation should be used to explore the disparities in program implementation by region. Going forward, the Evaluation Team recommends that program implementation fidelity be monitored formally to obtain reasonable consistency between regions. 3. Further study is required to determine whether the decline in program participation and effectiveness in the 2013-14 school year is an artifact of the substantial scaling-up of the program between 2012-13 and 2013-14. If the efficacy of the program remains diluted when implemented at scale, the program implementers should explore options such as: (1) Consortium for Educational Research and Evaluation North Carolina 23
making agreements with participating LEAs and schools to require attendance at institutes and professional development; (2) concentrating program resources on higher-intensity instructional coaching; and/or (3) limiting program size to ensure a high-intensity program. Consortium for Educational Research and Evaluation North Carolina 24
Appendix A. Evaluation Sample The Evaluation Team defined the evaluation sample for the 2012-13 academic year as all first- and second-year teachers who began receiving services by December 2012 and who worked in schools that were eligible for and agreed to participate in the (RttT schools). 12 This excludes teachers who began receiving supports in January 2013 or later, novice teachers receiving supports in non-rttt schools, the small number of third-year teachers served by the program in 2012-13, and TFA corps members. For the 2013-14 academic year, the Evaluation Team defined the evaluation sample as all first-, second-, and third-year teachers who began receiving supports by December 2013 and who worked in schools that were eligible for (RttT schools) and agreed to participate in the NC NTSP. While this includes third-year teachers, who were fully served by the program in 2013-14, this again excludes teachers who began receiving supports in January 2014 or later, novice teachers receiving supports in non-rttt schools, and TFA corps members. For this final RttT evaluation report, the Evaluation Team excluded these groups because (1) estimates of program performance need to be based upon a sample of teachers who received supports for a majority of the school year; (2) the primary objective of the Evaluation Team is to evaluate the performance of the program in its intended treatment area RttT schools; and (3) TFA corps members are significantly more likely to exit teaching after their two-year service commitment, are significantly more effective, on average, than other novice teachers (particularly in mathematics and science courses), and already receive induction services from TFA, and thus, did not fully participate in the. As shown in the left panel of Table A1 (following page), in the 2012-13 academic year the NC NTSP evaluation sample consisted of 344 teachers working in 59 schools and 16 LEAs. A large majority (72%) of this treatment sample were first-year teachers, and as stated above, all these teachers worked in RttT schools and entered the program by December 2012. The right panel of Table A1 shows that in the 2013-14 academic year the evaluation sample consisted of 808 teachers working in 91 schools and 25 LEAs. Nearly 50% of this treatment sample were first-year teachers, 36% were second-year teachers, and 15% were in their third year of teaching. All these teachers worked in RttT schools and entered the by December 2013. 12 The directive of the is to provide comprehensive induction services to novice teachers employed in the state s lowest-performing schools schools that in the year before RttT began were either in the lowest 5% of all schools in terms of student achievement or had graduation rates below 60%. We refer to these schools as RttT schools. Consortium for Educational Research and Evaluation North Carolina 25
Table A1. Evaluation Sample 2012-13 Evaluation Sample 2013-14 Evaluation Sample Teacher Count 344 808 Teachers by Region ECU 66 189 UNC-CSLD 80 237 UNCC 120 182 UNCG 78 200 1 st Year Teacher % 71.80 48.51 2 nd Year Teacher % 28.20 36.14 3 rd Year Teacher % n/a 15.35 Teachers in RttT Schools 344 808 Teachers Served by 344 808 Before January TFA Corps Members n/a n/a School Count 59 91 Schools by Region ECU 12 23 UNC-CSLD 15 29 UNCC 18 19 UNCG 14 20 LEA Count 16 25 LEAs by Region ECU 5 9 UNC-CSLD 4 5 UNCC 4 6 UNCG 3 5 To assess the impact of the, the Evaluation Team implemented a comparison group design to contrast the outcomes of teachers with those of other novice teachers working in low-performing schools. Isolating the effects of the on participating teachers is particularly challenging because several other RttT interventions were also concentrated in RttT schools over the same time period. The most notable was the District and School Transformation (DST) initiative, which provided regular professional development and coaching designed to improve the effectiveness and retention of teachers in RttT schools. To address these challenges, the Evaluation Team created two different comparison groups to better isolate the impact of the. For the first group, the Evaluation Team used school performance composite data from the 2011-12 school year to identify schools in the bottom decile of performance that were not eligible for (non-rttt schools) and did not receive services. In the 2012-13 school year this comparison group, labeled Non-RttT, consisted of all the first- and second-year teachers who began working in these schools by December 2012. In the 2013-14 school year this comparison group consisted of all the first-, Consortium for Educational Research and Evaluation North Carolina 26
second-, and third-year teachers who began working in these schools by December 2013. 13 Like the evaluation sample, this Non-RttT evaluation sample excludes TFA corps members and those beginning work in these schools after December. Overall, the left panel of Table A2 shows that the evaluation sample for the Non-RttT group consisted of 1,033 teachers working in 147 schools and 48 LEAs in the 2012-13 school year; in 2013-14, the Non-RttT evaluation sample consisted of 1,507 teachers working in 149 schools and 48 LEAs. For the second comparison group, the Evaluation Team identified RttT schools (eligible for the ) that did not participate in the. In the 2012-13 school year this comparison group, labeled Eligible, consisted of all the first- and second-year teachers who began working in these schools by December 2012. In the 2013-14 school year this comparison group consisted of all the first-, second-, and third-year teachers who began working in these schools by December 2013. As with the evaluation samples for the and the Non-RttT, the evaluation sample for the Eligible group excludes TFA corps members and those beginning work in these schools after December. Overall, the right panel of Table A2 shows that the evaluation sample for the Eligible group consisted of 201 teachers working in 32 schools and 18 LEAs in the 2012-13 school year. In 2013-14 the expanded to serve an additional set of RttT schools. Therefore, in the 2013-14 school year, the evaluation sample for the Eligible group was reduced in size and consisted of 169 teachers working in 16 schools and 11 LEAs. In the main body of the final evaluation report, the Evaluation Team only reports results for the Non-RttT sample; throughout the report appendices, the Evaluation Team provides results for both the Non-RttT and the Eligible groups. Table A2. Evaluation Sample Groups Non-RttT 2012-13 2013-14 Evaluation Evaluation Sample Sample Eligible 2012-13 2013-14 Evaluation Evaluation Sample Sample Characteristics Teacher Count 1033 1507 201 169 1 st Year Teacher % 60.47 40.69 60.70 39.05 2 nd Year Teacher % 39.53 34.77 39.30 33.14 3 rd Year Teacher % n/a 24.53 n/a 27.81 Teachers Hired Before 1033 1507 201 169 January TFA Corps Members n/a n/a n/a n/a School Count 147 149 32 16 LEA Count 48 48 18 11 Like the sample, both of these comparison groups consist of novice teachers working in low-performing schools. The Non-RttT group provides a larger sample for 13 Because the fully served third year teachers in the 2013-14 school year, the Evaluation Team included third year teachers in both comparison samples in 2013-14. Consortium for Educational Research and Evaluation North Carolina 27
analyses; however, since it is comprised of schools that did not receive RttT services, comparisons to this group do not allow for isolation of the impact of the from other RttT programs. The Eligible group addresses this concern by comparing teachers to novice teachers who received other RttT supports (primarily through DST). Because this group is much smaller, it provides less statistical power for detecting differences in outcomes. In addition, it is unknown why these schools declined to participate in the (in 2012-13 and 2013-14) and why some schools chose to enter the program in 2013-14. Therefore, analyses may not take into account other factors related to non-participation. Throughout subsequent appendices, the Evaluation Team presents results for the evaluation sample and the Non-RttT and Eligible evaluation samples. In appendices focused on teacher value-added to student achievement, teacher evaluation ratings, and teacher retention, the Evaluation Team presents two additional sets of results: (1) those for all teachers served by the and all novice teachers in comparison sample schools, and (2) those for an amended evaluation sample that includes teachers served in non-rttt schools who entered the by December of the academic year and were not TFA corps members. Table A3 (following page) presents school characteristics for the evaluation sample and for the evaluation samples for each of the comparison groups in 2012-13 and 2013-14. In 2012-13, teachers worked in schools with (1) more students qualifying for subsidized school meals; (2) more racial and ethnic minority students; (3) higher short-term suspension rates and higher violent acts rates (in comparison to Eligible schools); (4) higher percentages of novice teachers; and (5) lower performance composites (in comparison to the Non-RttT schools). In 2013-14, teachers worked in schools with (1) more students qualifying for subsidized school meals (in comparison to Non-RttT schools); (2) more racial and ethnic minority students; (3) higher short-term suspension rates (in comparison to Non-RttT schools); and (4) lower performance composites. Consortium for Educational Research and Evaluation North Carolina 28
Table A3. School Characteristics for the and Groups 2012-13 School Year 2013-14 School Year Non-RttT Eligible Non-RttT Group Group Group Eligible Group Characteristic Sample Sample Free and Reduced-Price Lunch Percentage 93.58 85.93 ** 87.16 + 91.78 87.40 ** 86.54 Racial/Ethnic Minority Percentage 90.87 78.37 ** 82.31 * 89.63 78.67 ** 80.55 + Short-Term Suspension Rate (Per 100 Students) 48.86 33.52 ** 27.56 ** 38.14 25.50 ** 26.30 Violent Acts Rate (Per 1000 Students) 12.24 10.96 4.82 ** 8.71 10.46 6.30 Total Per-Pupil Expenditures $11,001 $10,052 + $11,136 $10,534 $10,307 $10,214 Performance Composite 20.04 23.25 ** 22.55 31.07 34.29 ** 39.94 * Novice Teacher Percentage 32.99 27.19 ** 26.59 ** 32.93 30.29 32.63 School Level Elementary/Elementary- Middle Combination 35 (59.32%) 89 (60.54%) 23 (71.88%) 57 (62.64%) 94 (63.09%) 10 (62.50%) Middle School 10 24 2 13 23 1 (16.95%) (16.33%) (6.25%) (14.29%) (15.44%) (6.25%) High School 14 33 7 21 31 5 (23.73%) (22.45%) (21.88%) (23.08%) (20.81%) (31.25%) K-12 School 0 1 0 0 1 0 (0.00%) (0.68%) (0.00%) (0.00%) (0.67%) (0.00%) School Count 59 147 32 91 149 16 Note: This table displays school characteristics for schools in the, Non-RttT, and Eligible samples. +, *, and ** indicate statistically significant differences between schools and Non-RttT /NCNTSP Eligible schools at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 29
Appendix B. Data Sources The Evaluation Team collected the data presented in this final evaluation report from the UNC- GA implementation team, evaluation sample teachers and comparison sample teachers, and administrative datasets supplied by the North Carolina Department of Public Instruction (NCDPI). Specifically, this final evaluation report draws upon the following data sources: (1) participation records from each of the components of the institutes, instructional coaching, and professional development; (2) survey responses by evaluation sample teachers and Non-RttT sample teachers; and (3) student test scores, student demographics, classroom rosters, teacher evaluation ratings, certified salaries, school personnel, and school characteristics files provided by NCDPI. With these data sources the Evaluation Team examined levels of participation in components, teachers perceptions of program utility and self-efficacy/job satisfaction, and the impact of the on teacher value-added to student achievement, teacher evaluation ratings, and teacher retention. Below, we elaborate on the data sources used in this final evaluation report. Program Documents To examine the evaluation sample s participation in the components of the, the UNC- GA program implementers supplied the Evaluation Team with requested program documents. For the 2012-13 year this included (1) the curricula, agendas, and rosters of attendees for the 2012 Summer and Winter Institutes; (2) counts of instructional coach visits with each teacher; and (3) the agendas and rosters of attendees for professional development sessions. Data for the 2013-14 year included (1) the curricula, agendas, and rosters of attendees for the 2013 Regional Institutes and Statewide Institute; (2) counts of inperson and virtual instructional coach visits with each teacher; (3) contact hours inperson and virtual between teachers and program instructional coaches; and (4) the agendas and rosters of attendees for professional development sessions. Survey Responses by and Non-RttT Sample Teachers To assess (1) the perceptions of teachers regarding the focus and utility of components, and (2) the perceptions of both and Non-RttT sample teachers regarding school-provided novice teacher supports, school context, teacher practices, self-efficacy, and job satisfaction, the Evaluation Team partnered with North Carolina State University s (NCSU) College of Education to administer the Perceptions of Success Inventory Beginning Teachers (PSI-BT) survey in the spring of 2013 and 2014 (see the end of Appendix B for a complete survey). 14 For each survey administration the sample included and Non-RttT sample teachers in the evaluation sample. Both sets of teachers completed the regular PSI-BT items. In addition, teachers completed items developed by the Evaluation Team assessing the components institutes, instructional coaching, professional development of the. Researchers at NCSU used both online and paper-based mediums to administer the PSI-BT in spring 2013; in spring 2014 NCSU researchers administered the survey online only. 14 Please see http://ncsu.edu/succeed/beginning-teachers/ for more information on the development and use of the PSI-BT instrument. Consortium for Educational Research and Evaluation North Carolina 30
For the 2012-13 school year, the left panel of Table B1 presents response rates on the PSI-BT for the and Non-RttT samples overall, and for each region. Nearly half of the teachers in the evaluation sample responded to the survey (49.26%), with higher response rates for those in the ECU region. To encourage responses in the Non-RttT sample, the Evaluation Team provided a $5 financial incentive to survey completers; approximately 41% of Non-RttT sample teachers responded to the survey. The right panel of Table B1 displays PSI-BT response rates for the 2013-14 school year. Slightly more than half of the teachers in the evaluation sample responded to the survey (51.12%), with higher response rates for the ECU region and lower response rates for the UNC-CSLD region. As in 2012-13, the Evaluation Team offered a $5 financial incentive to Non- RttT sample teachers to complete the survey but only 17% responded in the 2013-14 school year. Table B1. PSI-BT Response Rates Survey Group 2012-13 PSI-BT Administration 2013-14 PSI-BT Administration Survey Respondents Administered Surveys Response Rate Survey Respondents Administered Surveys Response Rate Overall 167 339 49.26% 409 800 51.12% ECU 47 66 71.21% 147 187 78.61% UNC-CSLD 33 77 42.86% 78 236 33.05% UNCC 49 118 41.53% 88 179 49.16% UNCG 38 78 48.72% 96 198 48.48% Non-RttT Group 308 759 40.58% 179 1053 16.99% Due to the possibility for bias in the respondent sample, Table B2 (following page) displays data on participation in components for those teachers who did and did not respond to the survey (both overall and by region). In the 2012-13 school year, teachers who responded to the PSI-BT were more likely to attend an institute (66% for respondents versus 45% for non-respondents), received more instructional coaching visits per month (4.18 visits for respondents versus 3.33 for non-respondents), and attended more professional development sessions (2.87 PD sessions for respondents versus 1.60 PD sessions for non-respondents). In the 2013-14 school year, differences in program participation between PSI- BT respondents and non-respondents were reduced in magnitude; however, respondents were still more likely to attend an institute, to receive more instructional coaching visits, and to attend more professional development sessions. Given these differences, it is important to carefully interpret all survey results. Consortium for Educational Research and Evaluation North Carolina 31
Table B2. Participation in Components, by PSI-BT Response Status 2012-13 PSI-BT Administration 2013-14 PSI-BT Administration Region Overall Institute Attendance Non Respond Respond Average Instructional Coaching Visits Per Month Non Respond Respond Number of Professional Development Sessions Attended Non Respond Respond 66.47% 45.35% 4.18 3.33 2.87 1.60 ECU 80.85% 57.89% 6.35 5.20 2.89 1.21 UNC-CSLD 66.67% 70.45% 2.26 1.94 4.15 3.32 UNCC 67.35% 33.33% 3.53 3.31 1.86 0.67 UNCG 47.37% 32.50% 4.02 4.01 3.03 1.53 Overall 23.47% 17.14% 2.86 1.92 0.93 0.58 ECU 27.89% 25.00% 4.11 2.80 1.17 0.45 UNC-CSLD 21.79% 17.72% 1.81 1.59 1.60 1.04 UNCC 19.32% 13.19% 2.81 2.36 0.10 0.15 UNCG 21.88% 16.67% 1.87 1.70 0.79 0.27 Administrative Data from NCDPI To assess the impact of the on teacher value-added to student achievement, teacher evaluation ratings on the North Carolina Professional Teaching Standards (NCPTS), and teacher retention, the Evaluation Team used administrative datasets provided by NCDPI. Specifically, to estimate teacher value-added, the Evaluation Team used student test scores and demographics, classroom rosters, school personnel, and school characteristics files. This allows for the connection of students to their prior scores, demographics, teachers, and classroom peers; teachers to their characteristics; and students and teachers to the characteristics of their schools. To examine teacher evaluation ratings, the Evaluation Team used ratings on each of the five NCPTS that are directly assessed by school administrators. Finally, for teacher retention, the Evaluation Team used certified salary files to determine whether individuals returned to a teaching position overall, within the same LEA, or within the school school in North Carolina public schools. Consortium for Educational Research and Evaluation North Carolina 32
PSI-BT Survey Unless otherwise noted, PSI-BT items have a six-point scale: strongly disagree, disagree, slightly disagree, slightly agree, agree, and strongly agree. Domain One: School- or LEA-Provided Mentor Support Directions: Please respond to the survey questions below concerning the mentor provided by your SCHOOL or SCHOOL DISTRICT. 1) In my current experience in my school, a mentoring relationship is or would be important to me. 2) Has your school or school district assigned you a mentor? -If yes, continue answering mentoring questions -If no, skip to colleague support 3) In what month did you first meet with your school or school district assigned mentor? 1) July, 2) August, 3) September... 12) June 4) How much time do you spend meeting with your school or school district provided mentor each month? 1) 0-30 minutes, 2) 30 minutes to 1 hour, 3) 1-2 hours, 4) 2-3 hours, 5) 3-4 hours, 6) 4-5 hours, or 7) more than 5 hours 5) If you have a subject or grade-level specialty, does your school or school district assigned mentor teach the same subject or grade-level? 1) Yes, 2) No, or 3) Not applicable 6) My school or school district assigned mentor has provided assistance with classroom management. 7) My school or school district assigned mentor has provided assistance with instructional concerns. 8) My school or school district assigned mentor has provided assistance related to communication with parents or caregivers of my students. 9) My school or school district assigned mentor has provided assistance with unit or lesson planning. 10) My school or school district assigned mentor has provided assistance with analysis of student assessment data. 11) My school or school district assigned mentor is empathetic. 12) My school or school district assigned mentor encourages me to reflect about my teaching. Consortium for Educational Research and Evaluation North Carolina 33
13) Of the success you have had as a beginning teacher, what amount would you attribute to help from your school or school district assigned mentor? 1) a great deal, 2) quite a bit, 3) some, 4) hardly any, or 5) none 14) Overall, my school or school district assigned mentor has been helpful in developing my confidence, knowledge, and skills in teaching. Domain Two: Colleague Support Directions: Please respond to the survey questions below concerning the support you receive from colleagues in your school. 15) I have opportunities for meaningful conversations with other novice teachers in a setting free of evaluation. 16) I have common planning times with other teachers at my same grade level or subject area. 17) I have opportunities to visit and observe exemplary teachers. 18) I have a colleague in my same subject area or grade level who will answer my questions. Domain Three: Administration Support Directions: Please respond to the survey questions below concerning the support you receive from administrators in your school. 19) The administration at my school provides appropriate feedback for my discipline decisions. 20) The administration at my school encourages me to be an effective teacher. 21) The administration has oriented me to the school and staff. 22) I have on-going face-to-face communication with my administration. 23) The administration provides me with effective instructional leadership. Domain Four: Classroom Management Directions: Please respond to the survey questions below concerning your classroom management behaviors and success. 24) I have developed clear routines and procedures for my classroom that are aligned with school policy. 25) I have implemented consistent routines and procedures in my classroom. Consortium for Educational Research and Evaluation North Carolina 34
26) My routines and procedures positively impact the behavior of my students. 27) The discipline in my classroom is supportive of a good learning environment for my students. 28) I feel in control when I am teaching. 29) My students behaviors are consistent with my classroom expectations. 30) I am able to use communication to diffuse disruptive student behavior. Domain Five: Encouraging Student Success Directions: Please respond to the survey questions below concerning your instructional practices. 31) I am able to successfully teach students with a variety of ability levels. 32) I am able to motivate all students. 33) I am able to use a variety of teaching strategies to provide my students with instruction that is effective for them. 34) I am able to effectively teach my students from diverse backgrounds. 35) I am able to frame my instructional decisions based on my students learning. Domain Six: Curricular and Instructional Resources Directions: Please respond to the survey questions below concerning the curricular and instructional resources provided by your SCHOOL or SCHOOL DISTRICT. 36) I have adequate instructional supplies, such as copy paper, a functioning copier, and pens/pencils that I need for teaching. 37) I have been provided with curriculum that aligns with the state s objectives for my grade level or subject area. 38) I have the curriculum materials I need to teach effectively. 39) I have been provided with the instructional technology I need to teach effectively. 40) My students have the curricular resources they need to learn effectively. Consortium for Educational Research and Evaluation North Carolina 35
41) How many school or school district provided professional development sessions did you attend? 1) Zero, 2) 1-2, 3) 3-5, 4) 6-10, or 5) more than 10 42) The professional development provided by my school or school district improved my classroom management skills. 43) The professional development provided by my school or school district improved my ability to create rigorous, standards-aligned unit and lesson plans. 44) The professional development provided by my school or school district improved my ability to create rigorous, standards-driven assessments. 45) The professional development provided by my school or school district improved my ability to analyze student assessment data and adjust instruction. 46) The professional development provided by my school or school district improved my ability to reflect on my teaching practice. 47) The professional development provided by my school or school district provided me with instructional strategies that improved my students learning. 48) Overall, the professional development provided by my school or school district was helpful in developing my confidence, knowledge, and skills in teaching. Domain Seven: Assignment and Workload Directions: Please respond to the survey questions below concerning your current teaching assignments and workload. 49) I think the number of preparations I have for my classes is appropriate for a novice teacher. 50) I have at least one period per day that I can devote without interruption to planning for my classes. 51) My overall teaching workload is reasonable. 52) Novice teachers are allowed to choose whether to take on extra duties or not. 53) If this is your first year teaching, did you have a reduced teaching schedule or number of preparations? 1) Yes, 2) No, or 3) Not my first year teaching Consortium for Educational Research and Evaluation North Carolina 36
Domain Eight: Assessment and Evaluation Directions: Please respond to the survey questions below concerning how you assess and evaluate student learning. 54) I informally assess each student on a daily basis. 55) I formally assess each student on a weekly basis. 56) I use North Carolina s academic standards to create classroom assessments. 57) I am able to write quality unit/chapter tests for my students. 58) I am able to effectively create classroom assessments other than classroom tests. 59) I use a variety of measures to assess my students. 60) I use formative assessment in my classroom. 61) I use summative assessment in my classroom. 62) I have time to interpret my students assessment data. Domain Nine: Parental Contacts Directions: Please respond to the survey questions below concerning your contact with parents and/or caregivers. 63) The parents or caregivers of my students are supportive of their child s progress in school. 64) I am able to effectively communicate with my students parents or caregivers. 65) I have adequate guidance and support in working with parents or caregivers. Domain Ten: Satisfaction and Commitment Directions: Please respond to the survey questions below concerning your job satisfaction and commitment to teaching. 66) In general, I am satisfied with my current job. 67) I consider teaching to be my ideal career. 68) I feel inspired to instruct students to the best of my ability. Consortium for Educational Research and Evaluation North Carolina 37
69) If someone could change any of the following items, which ones would be the most important to improve your satisfaction with your job? Choose the THREE most important items only. 70) I am most interested in participating in and/or learning more about? 71) Think about your intentions regarding teaching. Which response best fits your current intentions? Domain Eleven: Demographics Please respond to the survey questions below concerning basic demographic information. 72) What level of teaching experience do you currently possess? 1) I am a first year teacher, 2) I am a second year teacher, 3) I am a third year teacher, or 4) I have taught for more than 3 years 73) What is the grade-level or subject-area your primarily teach? 1) PK, 2) K, 3) 1 st grade, 4) 2 nd grade, 5) 3 rd grade, 6) 4 th grade, 7) 5 th grade, 8) Middle school math, 9) Middle school reading, 10) Middle school science, 11) Middle school social studies, 12) High school math, 13) High school English, 14) High school science, 15) High school social studies, 16) Exceptional children, 17) PE/Health, 18) Art/music, or 19) Other 74) In what month of the 2012-13 school year did you begin teaching? 1) July, 2) August, 3) September... 12) June 75) I received my preparation to teach through: 1) Traditional four-year university program 2) Master of Arts in Teaching (MAT) 3) Licensure or certificate program 4) Lateral or alternative entry program 5) Other 76) I received my teacher training at: 1) North Carolina public university 2) North Carolina private university 3) Out-of-state university 4) On-line university/program 5) Teach For America 6) Other -Specific Items Added to the PSI-BT Unless otherwise noted, these items have a 6 point scale: strongly disagree, disagree, slightly disagree, slightly agree, agree, and strongly agree. Institutes 1) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to set ambitious academic goals for my students. Consortium for Educational Research and Evaluation North Carolina 38
2) After the North Carolina New Teacher Support Program Institute(s) I was better understood the Common Core Standards and North Carolina Essential Standards. 3) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to plan units and lessons to reach academic goals. 4) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to track student progress and adjust instruction. 5) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to implement instructional strategies that improve student learning. 6) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to create a respectful classroom environment that encourages student achievement. 7) After the North Carolina New Teacher Support Program Institute(s) I was better prepared for the school year. 8) Overall, the North Carolina New Teacher Support Program Institute(s) was helpful in developing my confidence, knowledge, and skills in teaching. Instructional Coaching 9) Has the North Carolina New Teacher Support Program assigned you an instructional coach? -If yes, continue answering instructional coaching questions -If no, skip to professional development questions 10) In what month did you first meet with the instructional coach provided by the North Carolina New Teacher Support Program? -Please provide a drop-down box of month choices, from July through June 11) How much time do you spend meeting with the instructional coach provided by the North Carolina New Teacher Support Program each month? -Please provide a drop-down box of the following time choices: 1) 0-30 minutes 2) 30 minutes to 1 hour 3) 1-2 hours 4) 2-3 hours 5) 3-4 hours 6) 4-5 hours 7) more than 5 hours 12) The virtual coaching provided by the North Carolina New Teacher Support Program effectively addressed my questions and concerns. -Responses: Strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly agree, AND Did not have virtual coaching 13) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance with classroom management. Consortium for Educational Research and Evaluation North Carolina 39
14) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance with instructional concerns. 15) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance related to communication with parents or caregivers of my students. 16) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance with unit or lesson planning. 17) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance with the Common Core State Standards and/or North Carolina Essential Standards. 18) My instructional coach assigned by the North Carolina New Teacher Support Program has provided assistance with analysis of student assessment data. 19) My instructional coach assigned by the North Carolina New Teacher Support Program is empathetic. 20) My instructional coach assigned by the North Carolina New Teacher Support Program encourages me to reflect about my teaching. 21) Of the success you have had as a beginning teacher, what amount would you attribute to help from your instructional coach assigned by the North Carolina New Teacher Support Program? -Please provide a drop-down box with the following choices: 1) None 2) Hardly any 3) Some 4) Quite a bit 5) A great deal 22) Overall, my instructional coach provided by the North Carolina New Teacher Support Program has been helpful in developing my confidence, knowledge, and skills in teaching. Professional Development 23) In your opinion, what factor would MOST improve attendance at North Carolina New Teacher Support Program professional development sessions? -Responses: 1) Providing a financial stipend 2) Holding professional development sessions during the school day; 3) Holding professional development sessions after school 4) Improving professional development curriculum; 5) Encouragement from schools/principals to attend; 6) Making the NTSP professional development your only required professional development (no longer have to attend school/district PD sessions) 24) Did you attend any North Carolina New Teacher Support Program professional development sessions? -If yes, proceed to professional development questions -If no, skip to PSI-BT questions Consortium for Educational Research and Evaluation North Carolina 40
25) The professional development provided by the North Carolina New Teacher Support Program improved my classroom management skills. 26) The professional development provided by the North Carolina New Teacher Support Program improved my ability to create rigorous, standards-aligned unit and lesson plans. 27) The professional development provided by the North Carolina New Teacher Support Program improved my ability to create rigorous, standards-driven assessments. 28) The professional development provided by the North Carolina New Teacher Support Program improved my ability to analyze student assessment data and adjust instruction. 29) The professional development provided by the North Carolina New Teacher Support Program improved my ability to reflect on my teaching practice. 30) The professional development provided by the North Carolina New Teacher Support Program provided me with instructional strategies that improved my students learning. 31) Overall, the professional development provided by the North Carolina New Teacher Support Program was helpful in developing my confidence, knowledge, and skills in teaching. Consortium for Educational Research and Evaluation North Carolina 41
Appendix C. To What Extent was the Implemented as it was Intended? As stated in North Carolina s RttT grant proposal, the directive of the is to provide comprehensive induction services to novice teachers employed in the state s lowest-performing schools schools that in the year before RttT began were either in the lowest 5% of all schools in terms of student achievement or had graduation rates below 60%. The program started with a limited implementation in the 2011-12 school year, serving 35 beginning teachers in 13 schools and five LEAs. Given this small sample of teachers and the delays in program implementation 15 the final evaluation report focuses on the 2012-13 and 2013-14 school years. As shown in the left panel of Table C1 (following page), in the 2012-13 academic year the NC NTSP scaled up to provide induction supports to 542 novice teachers in 77 schools and 19 LEAs. Over 70% of these program participants were first-year teachers, 25% were second-year teachers, and only a small percentage were in their third year of teaching. Ninety-four percent (510 out of 542) of these teachers worked in RttT schools; the remaining teachers worked in non- RttT schools who had sought out induction services for their beginning teachers. Seventy-seven percent (417 out of 542) of these teachers entered the before January 2013. Those who entered in January 2013 or after were either late hires and/or beginning teachers in schools that agreed to enter the in the spring of 2013. Finally, 7% (38 out of 542) of these teachers were Teach for America (TFA) corps members receiving additional induction services mentoring and professional development from TFA. The right panel of Table C1 (following page) indicates that in the 2013-14 academic year the NC NTSP scaled up again to provide induction supports to 1,108 novice teachers in 114 schools and 27 LEAs. Approximately 50% of these program participants were first-year teachers, 33% were second-year teachers, and 15% were in their third year of teaching. Eighty-one percent (903 out of 1,108) of these teachers worked in RttT schools; the remaining teachers worked in non-rttt schools who had sought out induction services for their beginning teachers. Over 93% (1,035 out of 1,108) of these teachers entered the before January 2014. Those who entered in January 2014 or after were predominantly late hires. Finally, 4% (47 out of 1,108) of these teachers were TFA corps members receiving additional induction services from TFA. Overall, the provided comprehensive induction supports to novice teachers in 73 RttT schools in the 2012-13 school year and 91 RttT schools in the 2013-14 school year. By the end of the RttT funding period, this represents over three-quarters of the lowest achieving schools to which the program was tasked to serve. Subsequent appendices display results for the evaluation sample only, for the complete sample of teachers (as shown in Table C1, following page), and for an amended evaluation sample for all non-tfa corps members who entered the program by December of the academic year. 15 The provided a summer institute in 2011 but did not provide instructional coaching and professional development until the spring of 2012. Consortium for Educational Research and Evaluation North Carolina 42
Table C1. To what extent was the implemented as it was intended? Characteristics 2012-13 School Year 2013-14 School Year Teacher Count 542 1108 Teachers by Region ECU 112 237 UNC-CSLD 157 348 UNCC 139 268 UNCG 134 255 1 st Year Teacher % 71.99 50.63 2 nd Year Teacher % 25.42 33.84 3 rd Year Teacher % 2.60 15.52 Teachers in RttT Schools 510 903 Teachers Served by Before January 417 1035 TFA Corps Members 38 47 School Count 77 114 Schools by Region ECU 19 30 UNC-CSLD 22 37 UNCC 18 24 UNCG 18 23 LEA Count 19 27 LEAs by Region ECU 6 10 UNC-CSLD 5 5 UNCC 4 7 UNCG 4 5 Consortium for Educational Research and Evaluation North Carolina 43
Appendix D. To What Extent did the Reach its Target Population? Institutes In advance of the 2012-13 school year, the held a week-long summer institute in Chapel Hill. Given the challenges of securing high attendance rates at a centralized institute in advance of the school year late hiring, conflicting LEA/school beginning teacher induction, and professional development the UNC-GA program implementers also offered a three-day, make-up winter institute in December 2012. As shown in the top panel of Table D1 (following page), 63 teachers in the evaluation sample attended the summer institute and 126 teachers (mutually-exclusive groups) attended the make-up winter institute. Of the 344 teachers in the 2012-13 evaluation sample, nearly 55% attended an institute. Across regions, however, there was sizable variation in attendance more than two-thirds of the evaluation sample teachers in the ECU and UNC-CSLD regions attended an institute while only 47% and 40% of the evaluation sample teachers from the UNCC and UNCG regions attended. In an effort to boost institute attendance in the 2013-14 school year, the UNC-GA program implementers offered two-day regional institutes in the ECU, UNC-CSLD, UNCC, and UNCG regions in advance of the school year (late July and early August) and a three-day, statewide institute in late September. As shown in the bottom panel of Table D1 (following page), 50 teachers in the evaluation sample attended a regional institute and 143 teachers (nonmutually exclusive groups) attended the statewide institute. Of the 808 teachers in the 2013-14 evaluation sample, only 21% attended any institute unlike 2012-13, institute attendance was fairly comparable across regions. Consortium for Educational Research and Evaluation North Carolina 44
Table D1. Evaluation Sample Attendance at Institutes 2012-13 School Year 2013-2014 School Year Institute Summer Institute Winter Institute Any Institute No Institute Overall Teacher Attendance 63 (18.31%) 126 (36.63%) 189 (54.94%) 155 (45.06%) Teacher Attendance by Region UNC- ECU Region CSLD Region UNCC Region UNCG Region 10 28 10 15 (15.15%) (35.00%) (8.33%) (19.23%) 39 25 46 16 (59.09%) (31.25%) (38.33%) (20.51%) 49 53 56 31 (74.24%) (66.25%) (46.67%) (39.74%) 17 27 64 47 (25.76%) (33.75%) (53.33%) (60.26%) Total Teachers 344 66 80 120 78 Regional Institute 50 16 12 11 11 (6.19%) (8.47%) (5.06%) (6.04%) (5.50%) Statewide 143 45 42 20 36 Institute (17.70%) (23.81) (17.72%) (10.99%) (18.00%) Any 166 52 45 30 39 Institute (20.54%) (27.51%) (18.99%) (16.48%) (19.50%) No Institute 642 137 192 152 161 (79.46%) (72.49%) (81.01%) (83.52%) (80.50%) Total Teachers 808 189 237 182 200 Instructional Coaching To assess the frequency of instructional coaching during the 2012-13 school year, the UNC-GA program implementers provided the Evaluation Team with counts of in-person coaching visits. Table D2 (third page following) displays these data by region and teacher experience (BT1=first-year teacher; BT2=second-year teacher). Overall, Table D2 conveys two main points. First, instructional coaches made frequent visits to the classrooms of the novice teachers they coached; however, the average number of visits varied substantially across regions. In the UNCC and UNCG regions, teachers typically averaged three to four in-person visits from their instructional coach each month approximately once a week. In the ECU region, teachers received more frequent instructional coaching, particularly early in the school year and at the start of the new semester, while in the UNC-CSLD region, teachers received approximately two visits per month once every other week. Second, across regions, first-year teachers typically received more in-person coaching visits than their second-year peers. For the 2013-14 school year, the UNC-GA program implementers made two changes to their instructional coaching and data collection efforts: (1) completing virtual instructional coaching visits with teachers, and (2) tracking the number of in-person and virtual contact hours spent with teachers. In the tables below for the 2013-14 school year, Table D3 (third page following) displays counts of in-person coaching visits with teachers, Table D4 Consortium for Educational Research and Evaluation North Carolina 45
(third page following) displays counts of virtual instructional coaching visits with teachers, and Table D5 (third page following) displays total contact hours in-person and virtual between teachers and their instructional coaches. Each of these tables displays data by region, overall, and by level of teacher experience. 16 In Table D3 (second page following), values for 2013-14 in-person instructional coaching visits indicate two key results. First, as in 2012-13, there was significant variation across regions in the average number of instructional coaching visits per teacher. On average, teachers in the ECU region received approximately three to four in-person visits per month, teachers in the UNCC region received slightly more than two visits per month, and teachers in the UNC-CSLD and UNCG regions received less than two visits per month. Second, on average, teachers received fewer in-person instructional coaching visits in 2013-14 than in 2012-13. For example, teachers in the ECU region often received four or more (particularly in August/September, October, and January) coaching visits per month in 2012-13, but in 2013-14 they averaged three to four visits per month. teachers in the UNCG region went from three to four visits per month in 2012-13 to less than two visits per month in 2013-14. Turning to virtual instructional coaching, Table D4 (third page following) shows that regions did not significantly rely upon virtual coaching to address novice teacher needs at most, teachers were averaging slightly more than one virtual coaching session per month. Similar to the in-person coaching numbers, substantial variation existed across regions. In the ECU region, teachers averaged approximately one virtual coaching session per month and in the UNCC region, teachers typically averaged between one-half and one virtual coaching session per month. By contrast, teachers in the UNC-CSLD and UNCG regions often averaged less than one-third or one-tenth of a virtual instructional coaching session per month; essentially, the instructional coaching in those two regions was almost entirely conducted through in-person, classroom visits. Table D5 (third page following) displays total contact hours, both in-person and virtual, between teachers and program instructional coaches in the 2013-14 school year. It should be noted that the UNC-GA program implementers switched to a new data management system early in the 2013-14 school year and as a result, contact hours for August and September are not available for (and thus not included in) the contact hours shown in Table D5. Like the substantial variation in instructional coaching visits shown in Tables D2-D4, Table D5 indicates that, on average, teachers in the ECU and UNCC regions experienced significantly more contact hours than their peers in the UNC-CSLD and UNCG regions. teachers in the ECU region averaged more than 33 in-person hours and nearly eight virtual contact hours throughout the 2013-14 school year; teachers in the UNCC region averaged nearly 29 in-person hours and five virtual contact hours throughout the school year. Over an eight month period from October through May this is approximately five hours of instructional coaching per month for teachers in the ECU region, and four hours of instructional coaching per month for teachers in the UNCC region. In contrast, NC NTSP teachers in the UNC-CSLD and UNCG regions averaged approximately ten in-person hours and less than one virtual contact hour throughout the school year; over an eight month 16 The UNC-GA program implementers switched to a new data management system early in the 2013-14 school year and as a result in-person and virtual instructional coaching data is not available for August and September. Consortium for Educational Research and Evaluation North Carolina 46
period from October to May, this is less than one and a half hours of instructional coaching per month. Across regions, these contact hours typically vary by level of teacher experience, with first-year teachers receiving the most instructional coaching and third-year teachers receiving the least amount of instructional coaching. For example, by level of experience, teachers in the UNCC region averaged 32, 28, and 21 in-person instructional coaching hours, respectively. Consortium for Educational Research and Evaluation North Carolina 47
Table D2. Average Number of Instructional Coaching Visits per Teacher (2012-13) ECU Region Visits UNC-CSLD Region Visits UNCC Region Visits UNCG Region Visits Month Overall BT 1 BT 2 Overall BT 1 BT 2 Overall BT 1 BT 2 Overall BT 1 BT 2 August & September 8.23 8.67 7.25 2.20 2.60 1.43 2.60 2.81 2.23 6.78 7.07 6.00 October 9.39 10.00 8.10 3.31 3.50 2.96 3.05 3.20 2.80 3.97 3.79 5.20 November 4.23 4.00 4.71 2.34 2.46 2.12 3.55 3.54 3.57 4.17 4.15 4.30 December 4.17 4.40 3.67 1.69 1.71 1.65 2.93 3.11 2.58 3.44 3.50 3.00 January 7.21 7.44 6.71 2.21 2.39 1.85 3.43 3.49 3.31 3.87 3.93 3.50 February 4.43 4.73 3.81 1.68 1.47 2.12 3.40 3.49 3.23 3.67 3.65 3.80 March 4.81 5.30 3.75 1.84 1.75 2.04 3.72 3.81 3.54 3.73 3.75 3.60 April 4.97 5.44 3.95 1.44 1.39 1.54 3.58 3.70 3.33 3.37 3.35 3.50 May & June 6.54 7.30 4.90 2.22 2.14 2.40 4.35 4.32 4.38 4.78 4.96 3.60 Table D3. Average Number of In-Person Instructional Coaching Visits per Teacher (2013-14) ECU Region Visits UNC-CSLD Region Visits UNCC Region Visits UNCG Region Visits Month Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 October 4.61 4.98 4.37 4.13 1.91 1.91 2.05 1.70 2.68 2.95 2.16 2.98 2.87 3.00 2.81 2.15 November 3.70 3.73 4.00 2.97 1.53 1.66 1.55 1.19 2.15 1.95 2.31 2.34 1.90 2.01 1.87 1.23 December 3.13 3.20 3.60 2.00 1.39 1.46 1.40 1.22 1.91 1.74 1.89 2.39 1.34 1.37 1.32 1.23 January 3.23 3.42 3.43 2.28 1.53 1.68 1.46 1.30 2.03 2.05 2.02 2.00 1.81 1.91 1.84 0.77 February 3.88 4.14 4.09 2.69 1.49 1.68 1.46 1.09 2.31 2.59 2.06 2.06 1.93 1.89 2.13 1.00 March 4.62 4.92 4.42 4.19 1.87 1.99 1.90 1.54 3.14 3.45 3.00 2.59 1.47 1.30 1.73 1.23 April 3.26 3.36 3.45 2.57 1.48 1.59 1.63 0.96 2.55 2.71 2.47 2.28 1.46 1.48 1.48 1.23 May & June 4.43 4.56 4.44 4.06 2.05 2.16 2.15 1.60 3.86 3.99 3.77 3.72 1.44 1.46 1.57 0.46 Consortium for Educational Research and Evaluation North Carolina 48
Table D4. Average Number of Virtual Instructional Coaching Visits per Teacher (2013-14) ECU Region Virtual Visits UNC-CSLD Region Virtual Visits UNCC Region Virtual Visits UNCG Region Virtual Visits Month Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 October 0.97 1.10 0.63 1.31 0.08 0.09 0.08 0.07 1.11 1.48 0.98 0.32 0.10 0.09 0.14 0.00 November 1.35 1.45 1.18 1.41 0.34 0.37 0.38 0.21 0.65 0.59 0.66 0.81 0.19 0.20 0.17 0.31 December 0.80 0.75 0.94 0.69 0.24 0.20 0.36 0.13 0.53 0.49 0.53 0.61 0.05 0.07 0.04 0.00 January 1.26 1.41 1.09 1.16 0.38 0.38 0.51 0.17 0.54 0.58 0.59 0.34 0.22 0.19 0.26 0.15 February 1.19 1.10 1.23 1.38 0.27 0.29 0.29 0.20 0.67 0.80 0.53 0.63 0.06 0.05 0.09 0.00 March 1.19 0.98 1.37 1.42 0.04 0.07 0.01 0.02 0.51 0.57 0.41 0.53 0.07 0.07 0.07 0.00 April 0.27 0.20 0.39 0.20 0.08 0.09 0.09 0.02 0.47 0.50 0.39 0.56 0.04 0.02 0.07 0.00 May & June 0.80 0.88 0.72 0.74 0.09 0.15 0.04 0.04 1.20 1.32 1.13 1.03 0.20 0.14 0.30 0.00 Table D5. Average Total Contact Hours with Instructional Coaches (2013-14) Region and Teacher Experience Average In-Person Hours Average Virtual Hours ECU Overall 33.49 7.67 ECU 1 st Year Teachers 36.34 7.79 ECU 2 nd Year Teachers 32.52 8.32 ECU 3 rd Year Teachers 27.26 6.00 UNC-CSLD Overall 10.38 0.97 UNC-CSLD 1 st Year Teachers 11.47 1.25 UNC-CSLD 2 nd Year Teachers 10.92 0.94 UNC-CSLD 3 rd Year Teachers 6.93 0.35 UNCC Overall 28.81 4.80 UNCC 1 st Year Teachers 32.28 4.42 UNCC 2 nd Year Teachers 27.92 4.66 UNCC 3 rd Year Teachers 21.27 6.10 UNCG Overall 9.70 0.50 UNCG 1 st Year Teachers 9.72 0.37 UNCG 2 nd Year Teachers 10.52 0.70 UNCG 3 rd Year Teachers 4.36 0.23 Consortium for Educational Research and Evaluation North Carolina 49
Professional Development For the 2012-13 school year, Table D6 presents the number of evaluation sample teachers in attendance at each of the professional development sessions (overall and by region). Table D7 displays the average number of professional development sessions attended per teacher, and the percentage of teachers attending zero, one, two, three, four, five, and six professional development sessions. Overall, Tables D6 and D7 indicate that within the evaluation sample (1) attendance at professional development sessions was low, with zero sessions attended as the modal value, and (2) significant heterogeneity existed across regions in attendance, with teachers in the UNC-CSLD region attending an average of 3.59 sessions (out of six) 17 and teachers in the UNCC region attending an average of 1.17 sessions (out of six). Table D6. Evaluation Sample Attendance at by PD Session and Region (2012-13) Region ECU UNC-CSLD UNCC UNCG Total PD Session 1 29 (43.94%) 57 (79.17%) 10 (8.47%) 23 (30.67%) 119 (35.95%) PD Session 2 30 (45.45%) 54 (72.97%) 27 (22.88%) 25 (32.05%) 136 (40.48%) PD Session 3 26 (39.39%) 50 (65.79%) 34 (28.81%) 25 (32.05%) 135 (39.94%) PD Session 4 27 (40.91%) 49 (61.25%) 27 (23.08%) 33 (42.31%) 136 (39.88%) PD Session 5 27 (42.86%) 42 (54.55%) 19 (16.52%) 33 (42.31%) 121 (36.34%) PD Session 6 20 (31.75%) 35 (45.45%) 23 (20.35%) 37 (47.44) 115 (34.74%) Table D7. Average Number of PD Sessions by Region for Evaluation Sample (2012-13) Avg. Number of PD Sessions Attended Number of PD Sessions Attended Overall Region ECU UNC-CSLD UNCC UNCG 2.22 2.41 3.59 1.17 2.26 Zero 34.01% 25.76% 12.50% 56.67% 28.21% One 14.24% 15.15% 8.75% 14.17% 19.23% Two 11.63% 16.67% 13.75% 6.67% 12.82% Three 10.76% 12.12% 13.75% 9.17% 8.97% Four 6.40% 4.55% 6.25% 6.67% 7.69% Five 11.63% 16.67% 13.75% 5.00% 15.38% Six 11.34% 9.09% 31.25% 1.67% 7.69% Teacher Count 344 66 80 120 78 17 The higher PD attendance values for the UNC-CSLD reflect the fact that several LEAs and schools in the region required their teachers to attend the PD sessions. Consortium for Educational Research and Evaluation North Carolina 50
For the 2013-14 school-year, Table D8 presents the number of evaluation sample teachers in attendance at each of the professional development sessions (by region and overall). Table D9 displays the average number of professional development sessions attended per teacher, and the percentage of teachers attending zero, one, two, three, four, five, and six professional development sessions. Overall, in comparison to the 2012-13 school year, attendance was much lower at the 2013-14 professional development sessions. During the 2012-2013 school year, evaluation sample teachers attended (on average) 2.22 professional development sessions, while the average number of sessions attended per teacher in 2013-14 was 0.76. Likewise, in 2012-13, 34% of evaluation sample teachers did not attend any program professional development sessions; in 2013-14, 71% of teachers did not attend any program professional development sessions. As in 2012-13, there were variations in attendance across regions, with teachers in the ECU and UNC-CSLD regions attending an average of 1.03 and 1.22 sessions (out of six) and teachers in the UNCC and UNCG regions attending an average of 0.14 and 0.52 sessions (out of six). Table D8. Evaluation Sample Attendance at by PD Session and Region (2013-14) Region ECU UNC-CSLD UNCC UNCG Total PD Session 1 32 (17.78%) 51 (22.77%) 9 (4.97%) 24 (12.50%) 116 (14.93%) PD Session 2 29 (15.59%) 59 (25.54%) 3 (1.65%) 19 (9.55%) 110 (13.78%) PD Session 3 42 (22.34%) 54 (22.98%) 5 (2.75%) 18 (9.05%) 119 (14.80%) PD Session 4 37 (20.00%) 39 (16.67%) 6 (3.30%) 16 (8.04%) 98 (12.25%) PD Session 5 41 (22.16%) 49 (20.94%) 2 (1.10%) 6 (3.03%) 98 (12.27%) PD Session 6 13 (7.07%) 38 (16.31%) 0 (0.00%) 21 (10.61%) 72 (9.03%) Table D9. Average Number of PD Sessions by Region for Evaluation Sample (2013-14) Avg. Number of PD Sessions Attended Number of PD Sessions Attended Overall Region ECU UNC-CSLD UNCC UNCG 0.76 1.03 1.22 0.14 0.52 Zero 70.92% 55.56% 65.40% 87.91% 76.50% One 10.52% 16.93% 3.80% 10.99% 12.00% Two 5.57% 11.11% 7.17% 0.55% 3.00% Three 4.95% 6.35% 7.17% 0.55% 5.00% Four 3.22% 7.41% 4.64% 0.00% 0.50% Five 2.48% 1.06% 6.75% 0.00% 1.00% Six 2.35% 1.59% 5.06% 0.00% 2.00% Teacher Count 808 189 237 182 200 Consortium for Educational Research and Evaluation North Carolina 51
Appendix E. How do Teachers Perceive the Impact of the Components on their Confidence, Knowledge, and Skills in Teaching? Institutes To what extent do teachers feel the Institutes positively impacted their teaching? To measure attendees perceptions of the utility of the institutes, the Evaluation Team developed a set of survey questions, aligned with the institute curriculum, asking attendees to indicate the extent to which they felt better prepared (after the institute) to carry out key teaching practices. The Evaluation Team included these items on the PSI-BT survey administered in 2013 and 2014 (see Appendix B). Table E1 presents the summative institute survey item, asking institute attendees the extent to which the institute(s) was helpful in developing their confidence, knowledge, and skills in teaching. Overall, the top panel of Table E1 shows that of those evaluation sample teachers who attended a 2012 institute and responded to the PSI-BT, 80% agreed or strongly agreed with the statement. This value varied from a high of 94% in the UNCC region to a low of 59% in the UNC-CSLD region. The bottom panel of Table E1 shows that 82% of the evaluation sample teachers who attended a 2013 institute and responded to the PSI-BT agreed or strongly agreed that the institute was helpful in developing their confidence, knowledge, and skills in teaching; this value varied from a high of 100% in the UNCG region to a low of 65% in the UNC-CSLD region. Table E1. Institute Summative Survey Item NTSP Instructional Response Coaching Item Groups 2012-13 Institutes Overall, the NC NTSP Institute(s) was helpful in developing my confidence, knowledge, and skills in teaching. 2013-14 Institutes Overall, the NC NTSP Institute(s) was helpful in developing my confidence, knowledge, and skills in teaching. n Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree Overall 110 2.73% 0.00% 2.73% 14.55% 42.73% 37.27% ECU 38 0.00% 0.00% 2.63% 15.79% 44.74% 36.84% UNC-CSLD 22 9.09% 0.00% 9.09% 22.73% 36.36% 22.73% UNCC 33 0.00% 0.00% 0.00% 6.06% 45.45% 48.48% UNCG 17 5.88% 0.00% 0.00% 17.65% 41.18% 35.29% Overall 97 1.03% 3.09% 1.03% 12.37% 43.30% 39.18% ECU 43 2.33% 4.65% 0.00% 9.30% 34.88% 48.84% UNC-CSLD 17 0.00% 5.88% 5.88% 23.53% 35.29% 29.41% UNCC 16 0.00% 0.00% 0.00% 25.00% 43.75% 31.25% UNCG 21 0.00% 0.00% 0.00% 0.00% 66.67% 33.33% Consortium for Educational Research and Evaluation North Carolina 52
Instructional Coaching To what extent do teachers feel instructional coaching positively impacted their teaching? To summatively assess the perceptions of evaluation sample teachers regarding the quality of instructional coaching, the Evaluation Team included the following items on the 2013 and 2014 PSI-BT surveys: 1. Overall, my (A) instructional coach and/or (B) school assigned mentor has been helpful in developing my confidence, knowledge, and skills in teaching. 2. Of the success you have had as a beginning teacher, what amount would you attribute to help from your (A) instructional coach and/or (B) school assigned mentor. teachers answered both A and B items; Non-RttT sample teachers answered B items only. Given the results shown in Appendix B, the Evaluation Team notes that extrapolating survey results to the full sample of and comparison sample teachers risks ignoring differences associated with non-response and should be done with caution. Overall, the top panel of Table E2 (following page) shows that during the 2012-13 school year, 78% of evaluation sample respondents agreed or strongly agreed that their program instructional coach was helpful in developing their confidence, knowledge, and skills in teaching. By comparison, 60% of evaluation sample teachers and 53% of Non-RttT sample teachers responded similarly regarding their school provided mentor. For the 2013-14 school year, the bottom panel of Table E2 indicates that 77% of evaluation sample respondents agreed or strongly agreed with the summative instructional coaching item. By comparison, 60% of evaluation sample teachers and 40% of Non-RttT sample teachers responded similarly regarding their school provided mentor. Each of these differences within the sample and between and Non-RttT sample teachers were statistically significant. Consortium for Educational Research and Evaluation North Carolina 53
Table E2. Summative Instructional Coaching/Mentoring Survey Items Survey Item Response Groups n Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree 2012-2013 School Year Overall, my IC provided by the NTSP has been helpful in developing my confidence, knowledge, and skills in teaching. Overall 165 5.45% 3.03% 1.21% 12.73% 33.94% 43.64% ECU 47 4.26% 4.26% 2.13% 19.15% 38.30% 31.91% UNC-CSLD 33 9.09% 6.06% 0.00% 12.12% 39.39% 33.33% UNCC 48 6.25% 2.08% 2.08% 14.58% 25.00% 50.00% UNCG 37 2.70% 0.00% 0.00% 2.70% 35.14% 59.46% Overall 159 7.55% 6.29% 5.66% 20.13% 32.08% 28.30% Overall, my school or school district assigned mentor has been helpful in developing my confidence, knowledge, and skills in teaching. ECU 42 11.90% 4.76% 2.38% 7.14% 38.10% 35.71% UNC-CSLD 33 9.09% 3.03% 15.15% 27.27% 30.30% 15.15% UNCC 47 8.51% 8.51% 2.13% 23.40% 29.79% 27.66% UNCG 37 0.00% 8.11% 5.41% 24.32% 29.73% 32.43% Group 297 10.77% 7.07% 5.05% 24.24% 28.28% 24.58% 2013-14 School Year Overall, my IC provided by the NTSP has been helpful in developing my confidence, knowledge, and skills in teaching. Overall 401 1.75% 3.24% 2.74% 15.21% 30.92% 46.13% ECU 144 0.00% 0.69% 4.17% 13.89% 25.69% 55.56% UNC-CSLD 74 2.70% 6.76% 2.70% 21.62% 33.78% 32.43% UNCC 87 4.60% 3.45% 2.30% 11.49% 35.63% 42.53% UNCG 96 1.04% 4.17% 1.04% 15.62% 32.29% 45.83% Overall 386 4.15% 8.29% 5.44% 22.02% 29.79% 30.31% Overall, my school or school district assigned mentor has been helpful in developing my confidence, knowledge, and skills in teaching. ECU 134 1.49% 10.45% 5.97% 28.36% 24.63% 29.10% UNC-CSLD 73 5.48% 4.11% 5.48% 13.70% 39.73% 31.51% UNCC 85 5.88% 10.59% 5.88% 25.88% 23.53% 28.24% UNCG 94 5.32% 6.38% 4.26% 15.96% 35.11% 32.98% Group 164 6.71% 13.41% 8.54% 31.10% 24.39% 15.85% Consortium for Educational Research and Evaluation North Carolina 54
Table E3. Summative Instructional Coaching/Mentoring Survey Items Survey Item Response Groups n None at All Hardly Any Some Quite a Bit Great Deal 2012-13 School Year Of the success you have had as a beginning teacher, what amount would you attribute to help from your IC assigned by the NTSP. Overall 165 0.61% 6.67% 30.30% 34.55% 27.88% ECU 47 0.00% 10.64% 36.17% 29.79% 23.40% UNC-CSLD 33 0.00% 6.06% 48.48% 33.33% 12.12% UNCC 48 2.08% 8.33% 22.92% 43.75% 22.92% UNCG 37 0.00% 0.00% 16.22% 29.73% 54.05% Overall 159 8.81% 22.01% 23.27% 26.42% 19.50% Of the success you have had as a beginning teacher, what amount would you attribute to help from your school or school district assigned mentor. ECU 42 11.90% 11.90% 21.43% 28.57% 26.19% UNC-CSLD 33 18.18% 30.30% 24.24% 21.21% 6.06% UNCC 47 4.26% 25.53% 21.28% 25.53% 23.40% UNCG 37 2.70% 21.62% 27.03% 29.73% 18.92% Group 298 10.74% 19.13% 27.52% 28.19% 14.43% 2013-14 School Year Of the success you have had as a beginning teacher, what amount would you attribute to help from your IC assigned by the NTSP. Overall 401 3.99% 8.48% 29.43% 30.67% 27.43% ECU 144 1.39% 6.94% 25.69% 29.17% 36.81% UNC-CSLD 74 8.11% 14.86% 24.32% 35.14% 17.57% UNCC 87 5.75% 8.05% 33.33% 29.89% 22.99% UNCG 96 3.12% 6.25% 35.42% 30.21% 25.00% Overall 387 5.68% 13.44% 31.78% 25.32% 23.77% Of the success you have had as a beginning teacher, what amount would you attribute to help from your school or school district assigned mentor. ECU 134 2.99% 16.42% 34.33% 24.63% 21.64% UNC-CSLD 72 8.33% 6.94% 31.94% 27.78% 25.00% UNCC 87 8.05% 17.24% 29.89% 20.69% 24.14% UNCG 94 5.32% 10.64% 29.79% 28.72% 25.53% Group 164 12.20% 23.17% 35.37% 20.12% 9.15% The top panel of Table E3 indicates that in the 2012-13 school year, 62% of evaluation sample respondents attributed quite a bit or a great deal of their teaching success to help from their instructional coaches. By comparison, 46% of evaluation Consortium for Educational Research and Evaluation North Carolina 55
sample teachers and 43% of Non-RttT sample teachers responded similarly regarding their school provided mentor. For the 2013-14 school year, the bottom panel of Table E3 (previous page) shows that 58% of evaluation sample respondents attributed quite a bit or a great deal of their teaching success to help from their instructional coaches. By comparison, 49% of evaluation sample teachers and 29% of Non-RttT sample teachers responded similarly regarding their school provided mentor. Each of these differences within the sample and between and Non-RttT sample teachers were statistically significant. Professional Development To what extent do teachers feel the professional development positively impacted their teaching? To summatively assess the perceptions of evaluation sample teachers regarding the quality of professional development sessions, the Evaluation Team included the following item on the 2013 and 2014 PSI-BT surveys: Overall, the professional development provided by (A) the and/or (B) my school or school district has been helpful in developing my confidence, knowledge, and skills in teaching. teachers answered both A and B items; Non-RttT sample teachers answered the B item only. Unfortunately, a problem with the 2013-14 administration of the PSI-BT survey responses for agree and disagree took on the same response value prevents the Evaluation Team from presenting the 2013-14 results. Given the results shown in Appendix B, the Evaluation Team notes that extrapolating survey results to the full sample of and comparison sample teachers risks ignoring differences associated with non-response and should be done with caution. Overall, the top panel of Table E4 (following page) shows that in the 2012-13 school year 87% of evaluation sample teachers who attended professional development sessions and responded to the PSI-BT agreed or strongly agreed with the survey item. By comparison, 66% of evaluation sample respondents answered similarly for their school provided professional development; 60% of Non-RttT sample respondents answered similarly for their school provided professional development. Each of these differences within the sample and between and Non-RttT sample teachers were statistically significant. Consortium for Educational Research and Evaluation North Carolina 56
Table E4. Summative Professional Development Survey Item Survey Item Overall, the PD provided by the was helpful in developing my confidence, knowledge, and skills in teaching. Overall, the PD provided by my school or school district was helpful in developing my confidence, knowledge, and skills in teaching. Response Groups n Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree Overall 132 0.76% 0.00% 1.52% 10.61% 45.45% 41.67% ECU 39 0.00% 0.00% 0.00% 12.82% 53.85% 33.33% UNC-CSLD 26 3.85% 0.00% 0.00% 15.38% 50.00% 30.77% UNCC 32 0.00% 0.00% 3.12% 9.38% 50.00% 37.50% UNCG 35 0.00% 0.00% 2.86% 5.71% 28.57% 62.86% Overall 164 4.27% 0.00% 6.71% 23.17% 50.61% 15.24% ECU 47 0.00% 0.00% 8.51% 19.15% 59.57% 12.77% UNC-CSLD 32 15.62% 0.00% 3.12% 25.00% 34.38% 21.88% UNCC 48 2.08% 0.00% 8.33% 31.25% 47.92% 10.42% UNCG 37 2.70% 0.00% 5.41% 16.22% 56.76% 18.92% Group 305 5.25% 0.00% 7.54% 27.21% 46.23% 13.77% Consortium for Educational Research and Evaluation North Carolina 57
Appendix F. How do Teachers Perceive the Impact of the on their Self-Efficacy and Job Satisfaction? To measure perceptions of self-efficacy and job satisfaction for and Non-RttT sample teachers, the Evaluation Team relied on a set of pre-existing items on the PSI-BT survey. Table F1 (following page) displays these results for the 2012-13 school year and Table F2 (second page following) presents these results for the 2013-14 school year. Like the survey responses presented earlier, due to potential non-response bias these results should be interpreted with caution. In terms of self-efficacy for the 2012-13 school year, 79% of evaluation sample respondents (versus 73% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the statement that they can successfully instruct students with a variety of ability levels; 60% of evaluation sample respondents (versus 54% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the statement that they can motivate all students; and 84% of evaluation sample respondents (versus 78% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the statement that they feel inspired to instruct students to the best of their ability. For the third selfefficacy item ( I feel inspired to instruct students to the best of my ability ), the difference between and Non-RttT sample teachers was statistically significant. Regarding job satisfaction, 59% of evaluation sample respondents (versus 55% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the statement that they were satisfied with their current job, and 71% of evaluation sample respondents (versus 67% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the statement that they considered teaching to be their ideal career. Consortium for Educational Research and Evaluation North Carolina 58
Table F1. Self-Efficacy and Job Satisfaction Questions (2012-13) Efficacy and Job Satisfaction Items I am able to successfully teach students with a variety of ability levels. I am able to motivate all students. I feel inspired to instruct students to the best of my ability. In general, I am satisfied with my current job. I consider teaching to be my ideal career. Response Strongly Slightly Slightly Strongly Groups n Disagree Disagree Disagree Agree Agree Agree Overall 166 0.00% 1.81% 2.41% 16.87% 62.05% 16.87% ECU 47 0.00% 2.13% 2.13% 17.02% 63.83% 14.89% UNC-CSLD 33 0.00% 0.00% 0.00% 27.27% 60.61% 12.12% UNCC 49 0.00% 2.04% 4.08% 12.24% 65.31% 16.33% UNCG 37 0.00% 2.70% 2.70% 13.51% 56.76% 24.32% Comp. Group 307 0.33% 1.63% 2.28% 23.13% 55.05% 17.59% Overall 166 0.00% 6.02% 7.83% 25.90% 47.59% 12.65% ECU 47 0.00% 4.26% 0.00% 29.79% 55.32% 10.64% UNC-CSLD 33 0.00% 6.06% 15.15% 18.18% 45.45% 15.15% UNCC 49 0.00% 8.16% 10.20% 28.57% 44.90% 8.16% UNCG 37 0.00% 5.41% 8.11% 24.32% 43.24% 18.92% Comp. Group 307 1.95% 4.89% 11.73% 27.36% 41.37% 12.70% Overall 164 3.05% 1.83% 4.88% 6.10% 36.59% 47.56% ECU 46 2.17% 4.35% 6.52% 8.70% 36.96% 41.30% UNC-CSLD 32 3.12% 0.00% 6.25% 3.12% 40.62% 46.88% UNCC 49 4.08% 2.04% 2.04% 2.04% 36.73% 53.06% UNCG 37 2.70% 0.00% 5.41% 10.81% 32.43% 48.65% Comp. Group 308 0.97% 1.62% 6.17% 13.31% 34.74% 43.18% Overall 164 7.93% 5.49% 7.93% 19.51% 34.15% 25.00% ECU 46 6.52% 6.52% 2.17% 19.57% 36.96% 28.26% UNC-CSLD 32 9.38% 9.38% 15.62% 25.00% 21.88% 18.75% UNCC 49 8.16% 6.12% 0.00% 22.45% 34.69% 28.57% UNCG 37 8.11% 0.00% 18.92% 10.81% 40.54% 21.62% Comp. Group 308 6.82% 7.79% 10.71% 20.13% 36.69% 17.86% Overall 164 4.27% 3.66% 4.27% 17.07% 40.24% 30.49% ECU 46 2.17% 4.35% 4.35% 26.09% 32.61% 30.43% UNC-CSLD 32 6.25% 0.00% 6.25% 9.38% 43.75% 34.38% UNCC 49 6.12% 4.08% 2.04% 14.29% 42.86% 30.61% UNCG 37 2.70% 5.41% 5.41% 16.22% 43.24% 27.03% Comp. Group 308 3.57% 5.84% 5.84% 17.53% 32.14% 35.06% Turning to the 2013-14 school year (Table F2, following page), responses to the same three selfefficacy survey items indicate that 84%, 69%, and 82% of evaluation sample respondents (versus 74%, 55%, and 75% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the survey items. Regarding job satisfaction, responses to the same two survey items show that 57% and 68% of evaluation sample respondents (versus 48% and 61% of Non-RttT evaluation sample respondents) agreed or strongly agreed with the survey items. For all five of these self-efficacy and job satisfaction items, the differences between and Non-RttT sample teachers Consortium for Educational Research and Evaluation North Carolina 59
were statistically significant. These differences must be interpreted carefully, however, due to non-response bias and the low response rate for the Non-RttT sample teachers. Table F2. Self-Efficacy and Job Satisfaction Questions (2013-14) Survey Items I am able to successfully teach students with a variety of ability levels. I am able to motivate all students. I feel inspired to instruct students to the best of my ability. In general, I am satisfied with my current job. I consider teaching to be my ideal career. Response Strongly Slightly Slightly Strongly Groups n Disagree Disagree Disagree Agree Agree Agree Overall 408 0.00% 0.49% 1.72% 13.73% 57.11% 26.96% ECU 147 0.00% 0.68% 1.36% 14.97% 57.14% 25.85% UNC-CSLD 77 0.00% 1.30% 1.30% 15.58% 54.55% 27.27% UNCC 88 0.00% 0.00% 3.41% 12.50% 52.27% 31.82% UNCG 96 0.00% 0.00% 1.04% 11.46% 63.54% 23.96% Comp. Group 178 0.56% 1.69% 3.93% 19.66% 52.81% 21.35% Overall 408 1.23% 3.68% 6.37% 19.85% 45.59% 23.28% ECU 147 0.68% 3.40% 6.80% 15.65% 50.34% 23.13% UNC-CSLD 77 1.30% 6.49% 11.69% 14.29% 40.26% 25.97% UNCC 88 2.27% 4.55% 6.82% 30.68% 37.50% 18.18% UNCG 96 1.04% 1.04% 1.04% 20.83% 50.00% 26.04% Comp. Group 178 2.81% 2.81% 11.80% 28.09% 39.89% 14.61% Overall 406 1.48% 1.97% 1.97% 13.05% 35.71% 45.81% ECU 146 2.05% 2.05% 1.37% 11.64% 37.67% 45.21% UNC-CSLD 78 2.56% 2.56% 1.28% 10.26% 35.90% 47.44% UNCC 87 1.15% 3.45% 3.45% 16.09% 37.93% 37.93% UNCG 95 0.00% 0.00% 2.11% 14.74% 30.53% 52.63% Comp. Group 177 2.26% 4.52% 2.82% 15.25% 31.07% 44.07% Overall 407 7.13% 8.35% 6.63% 21.13% 35.38% 21.38% ECU 147 4.76% 6.80% 6.80% 24.49% 34.69% 22.45% UNC-CSLD 78 11.54% 10.26% 6.41% 19.23% 35.90% 16.67% UNCC 87 8.05% 12.64% 6.90% 22.99% 32.18% 17.24% UNCG 95 6.32% 5.26% 6.32% 15.79% 38.95% 27.37% Comp. Group 178 12.36% 10.11% 11.80% 17.98% 35.39% 12.36% Overall 404 3.47% 5.69% 4.46% 18.32% 32.67% 35.40% ECU 146 3.42% 4.79% 4.79% 18.49% 30.82% 37.67% UNC-CSLD 78 3.85% 5.13% 5.13% 20.51% 23.08% 42.31% UNCC 85 4.71% 7.06% 1.18% 20.00% 42.35% 24.71% UNCG 95 2.11% 6.32% 6.32% 14.74% 34.74% 35.79% Comp. Group 178 4.49% 8.43% 6.74% 19.66% 27.53% 33.15% Consortium for Educational Research and Evaluation North Carolina 60
Appendix G. To What Extent does the Impact Teacher Effectiveness as Measured by Teacher Value-Added to Student Achievement? EVAAS Results For analyses of teacher value-added to student achievement, the Evaluation Team presents four sets of results: (1) overall results for the 2012-13 school year, the 2013-14 school year, and the two years combined; (2) regional results (separate results for teachers in the ECU, UNC-CSLD, UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14 school year, and the two years combined; (3) cohort results (separate results for Cohort 1 teachers entering the program in the 2012-13 school year and Cohort 2 teachers entering the program in 2013-14) for the 2012-13 school year, the 2013-14 school year, and the two years combined; 18 and (4) first-year teacher results, which focus on first-year teachers in the 2012-13 school year, the 2013-14 school year, and the two years combined. 19 Taken together, this set of analyses presents evidence regarding the overall effectiveness of the program and how that effectiveness may have differed by region, cohort, and for first-year teachers. These sub-analyses are particularly important given the differences in treatment components across regions and cohorts and the hypothesis of stronger program effects for first-year teachers. To assess the contributions of teachers to student achievement, the Evaluation Team analyzed EVAAS teacher effectiveness estimates generated by the SAS Institute. The state uses these estimates for assessing teacher effectiveness for Standard 6 of the North Carolina Educator Evaluation System. For these models, the Evaluation Team made teacher EVAAS estimates the outcome variable and regressed this value-added measure on a set of school characteristics and, when applicable, teacher experience indicators and year fixed effects. The Evaluation Team chose to control for these variables due to the significant differences in school characteristics between and comparison sample schools and because these contextual variables are not accounted for in EVAAS models. The Evaluation Team clustered standard errors at the school level to account for dependence in the data. Results from these models express the adjusted-average differences in student achievement in normal curve equivalency units between students taught by teachers and students taught by Non-RttT or Eligible sample teachers. Like the value-added models in the main body of the report, the Evaluation Team combined EVAAS data from elementary and middle grades and ran separate models for mathematics, reading, and science (End-of-Grade (EOG) science exams in grades five and eight). Additionally, the Evaluation Team combined EVAAS data for End-of-Course (EOC) exams (Biology, English II, and Math I) and ran a single EOC model. Tables G1 (third page following) and G2 (fourth page following) display overall and regional EVAAS results for elementary and middle grades and EOC exams. Overall, the top panel of Table G1 shows that in elementary and middle grades mathematics, students taught by 18 Cohort 1 teachers (evaluation sample) include first- and second-year teachers. Cohort 2 teachers (evaluation sample) include first-, second-, and third-year teachers. 19 There are a small number of teachers who have zero years of experience in both the 2012-13 and 2013-14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from the first-year teacher sample in 2013-14. Consortium for Educational Research and Evaluation North Carolina 61
teachers made significantly larger achievement gains than students taught by Non-RttT and Eligible sample teachers in the 2012-13 school year. There were no significant mathematics differences in 2013-14 or using two years of pooled data. Like mathematics, in elementary and middle grades reading, students taught by teachers made significantly larger achievement gains than students taught by Non-RttT and Eligible sample teachers in 2012-13. teachers were less effective than Eligible sample teachers in 2013-14 and there were no statistically significant differences with two years of pooled data. For fifth and eighth grade science, students taught by teachers made significantly larger achievement gains than students taught by Non-RttT sample teachers in 2013-14 and with two years of pooled data. Examining these overall results by region, the middle and bottom panels of Table G1 indicate that students taught by teachers in the ECU region made larger achievement gains, across subjects (mathematics, reading, and science), than students taught by teachers in either comparison group. teachers in the UNC-CSLD and UNCC regions were more effective in mathematics and reading than teachers in either comparison group in the 2012-13 school year; teachers in the UNC-CSLD region were less effective than NC NTSP Eligible sample teachers in reading in the 2013-14 school year. teachers in the UNCG region were less effective than Eligible sample teachers in reading in 2013-14 and less effective than both comparison groups in fifth and eighth grade science. The top panel of Table G2 shows that for EOC exams in 2013-14, students taught by teachers made significantly smaller achievement gains than students taught by Eligible sample teachers. Examining the EVAAS EOC results by region, the middle and bottom panels of Table G2 indicate teachers from the ECU region were less effective than both comparison groups in 2013-14 and with two years of pooled data. NC NTSP teachers from the UNC-CSLD and UNCC regions were less effective than Eligible sample teachers in 2013-14. teachers in the UNCG region were more effective than both comparison groups in 2012-13 and more effective than the Non-RttT sample with two years of pooled data. To determine how these EVAAS estimates may differ by cohort, Tables G3 (fourth page following) and G4 (fourth page following) present results for elementary and middle grades and EOC exams. For the 2012-13 school year, results in the top and bottom panels of Table G3 repeat results from Table G1: Cohort 1 teachers were more effective in mathematics and reading. Examining the 2013-14 school year scores, where data are available for both Cohort 1 and Cohort 2 teachers, results show that Cohort 2 teachers were less effective than Eligible sample teachers in reading; furthermore, NC NTSP teachers in both cohorts were more effective than Non-RttT sample teachers in fifth and eighth grade science. With two years of pooled data, Cohort 1 teachers were more effective than Non-RttT sample teachers in mathematics and reading. For EOC exams (shown in Table G4), cohort results in 2012-13 match the overall results in Table G2. Results in 2013-14 show that teachers in both cohorts were less effective than Eligible sample teachers; there were no significant differences with two years of pooled data. Consortium for Educational Research and Evaluation North Carolina 62
Finally, Tables G5 (fourth page following) and G6 (fourth page following) present EVAAS results for first-year teachers only. For the 2012-13 school year, Table G5 shows that students taught by first-year teachers made significantly larger achievement gains in mathematics and reading than students taught by both sets of comparison sample teachers; Table G6 indicates that first-year teachers were more effective than first-year Non-RttT sample teachers on EOC exams. In 2013-14, there were no significant differences for first-year teachers in mathematics, reading, or fifth and eighth grade science, but teachers were less effective than first-year Eligible sample teachers on EOC exams. Using two years of pooled data, first-year teachers were more effective than both comparison groups in elementary and middle grades mathematics, more effective than Eligible sample teachers in elementary and middle grades reading, and more effective than Non-RttT sample teachers in fifth and eighth grade science. Overall, these EVAAS results are comparable to the student-level value-added results presented later in this appendix. Evidence indicates that teachers were more effective than comparison sample teachers in elementary and middle grades. These results were concentrated in 2012-13 (Cohort 1 teachers) and did not extend (outside of fifth and eighth grade science) into the 2013-14 school year. Conversely, in the 2013-14 school year, teachers were less effective in multiple comparisons with Eligible sample teachers. Consortium for Educational Research and Evaluation North Carolina 63
Table G1. Overall and Regional EVAAS Results (Evaluation Sample) Group vs. Non-RttT vs. Eligible Elementary & Middle Grades Math Elementary & Middle Grades Reading 5 th & 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 3.054 ** 0.155 0.828 1.670 ** 0.015 0.259 0.971 1.239 * 1.011 + 3.923 ** -1.152 1.294 2.012 ** -0.917 ** 0.229 0.665-0.043 0.498 Regions vs Non-RttT ECU Region 5.623 ** 1.401 2.219 ** 2.582 ** 0.774 * 1.116 ** 4.047 * 2.992 + 2.772 * UNC-CSLD Region 4.465 * -1.165-0.007 2.443 ** -0.733-0.161 1.892 0.833 1.052 UNCC Region 3.041 * 1.727 1.915 + 1.130 + 0.090 0.236-0.868 1.412 0.234 UNCG Region -0.923-0.915-1.033 0.295-0.266-0.336-1.578 + 0.221-0.051 Regions vs Eligible ECU Region 6.140 ** 0.068 2.623 * 2.872 ** -0.160 1.055 3.477 + 1.357 2.053 UNC-CSLD Region 4.982 * -2.499 0.396 2.733 ** -1.668 ** -0.221 1.322-0.802 0.334 UNCC Region 3.558 * 0.393 2.318 + 1.420 + -0.845 0.175-1.438-0.222-0.485 UNCG Region -0.407-2.248-0.629 0.585-1.200 * -0.397-2.148 + -1.414 + -0.769 Note: The top panel of this table displays differences in adjusted-average EVAAS estimates for teachers and Non-RttT and Eligible sample teachers. The second panel of this table displays differences in adjusted-average EVAAS estimates for teachers, by region, and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates for teachers, by region, and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 64
Table G2. Overall and Regional EVAAS Results (Evaluation Sample) Group EOC Exams 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 vs. Non-RttT 0.693-0.106 0.083 vs. Eligible 0.455-1.812 ** -0.464 Regions vs Non-RttT ECU Region 0.300-1.361 ** -0.652 * UNC-CSLD Region --- -0.843-0.727 UNCC Region -0.469-0.444-0.370 UNCG Region 1.919 * 1.112 1.214 * Regions vs Eligible ECU Region 0.120-2.727 ** -1.047 ** UNC-CSLD Region --- -2.208 + -1.122 UNCC Region -0.649-1.809 * -0.765 UNCG Region 1.739 + -0.253 0.819 Note: The top panel of this table displays differences in adjusted-average EVAAS estimates for teachers and Non-RttT and Eligible sample teachers. The second panel of this table displays differences in adjusted-average EVAAS estimates for teachers, by region, and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates for teachers, by region, and the reference group of NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 65
Table G3. Cohort EVAAS Results (Evaluation Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 Cohorts vs Non-RttT Cohort 1 3.054 ** 0.639 1.483 + 1.670 ** 0.307 0.759 * 0.971 1.208 + 1.127 Cohort 2 --- -0.189-0.011 --- -0.183-0.273 --- 1.261 + 0.862 Cohorts vs Eligible Cohort 1 3.923 ** -0.675 1.860 2.012 ** -0.617 0.679 0.665-0.071 0.581 Cohort 2 --- -1.503 0.366 --- -1.107 ** -0.353 --- -0.018 0.316 Note: The top panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 NC NTSP teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G4. Cohort EVAAS Results (Evaluation Sample) Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams Cohorts vs Non-RttT Cohort 1 0.693-0.088 0.029 Cohort 2 --- -0.122 0.202 Cohorts vs Eligible Cohort 1 0.455-1.790 * -0.530 Cohort 2 --- -1.823 * -0.356 Note: The top panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 66
Table G5. First-Year EVAAS Results (Evaluation Sample) Group vs. Non-RttT vs. Eligible Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2.985 * 0.161 1.442 + 1.331 ** -0.341 0.115 1.211 1.183 1.442 + 5.974 ** -1.172 3.472 * 2.389 ** -0.425 0.943 + 1.741-0.070 1.430 Note: This table displays differences in adjusted-average EVAAS estimates between first-year teachers and first-year Non-RttT and NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G6. First-Year EVAAS Results (Evaluation Sample) Group 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams vs. Non-RttT 1.304 * -0.047 0.596 vs. Eligible 0.775-2.922 ** -0.341 Note: This table displays differences in adjusted-average EVAAS estimates between first-year teachers and first-year Non-RttT and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 67
Student-Level Value-Added Results As an alternative way to assess the contributions of teachers to student achievement, the Evaluation Team estimated ordinary least squares (OLS) regression models with students standardized test scores (standardized within subject, grade, and year for End-of-Grade exams and subject and year for End-of-Course exams) as the outcome variable, an extensive set of student, classroom, teacher, and school characteristics to help isolate the impact of the on adjusted-average student achievement gains, and standard errors clustered at the school level to account for dependence in the data. Results from these models express the adjusted-average differences in student achievement expressed in standard deviations of student achievement between students taught by teachers and students taught by Non-RttT or Eligible sample teachers. Given the relatively small sample of teachers from the and the comparison groups who taught a tested grade/subject, the Evaluation Team combined data from elementary and middle grades (four to eight) and ran separate models for mathematics, reading, and science (EOG science exams in grades five and eight). Additionally, the Evaluation Team combined data for the three high school (grades nine through twelve) EOC exams (Biology, English II, and Math I) and ran a single high school EOC model. In this appendix, the Evaluation Team displays results for the evaluation sample and for (1) the complete sample of teachers served by the and in the comparison groups, and (2) an amended evaluation sample that includes teachers who worked in non-rttt schools still excluding TFA corps members and those entering the in January or later. Tables G7 (second page following) and G8 (third page following) display overall and regional value-added results for elementary and middle grades and high school. Overall, the top panel of Table G7 shows that in elementary and middle grades mathematics, students taught by teachers made significantly larger achievement gains than students taught by Non- RttT and Eligible sample teachers in the 2012-13 school year. There were no significant mathematics differences in 2013-14; using two years of pooled data, teachers were more effective than Non-RttT sample teachers. Like mathematics, in elementary and middle grades reading, students taught by teachers made significantly larger achievement gains than students taught by Non-RttT and Eligible sample teachers during the 2012-13 school year. For fifth and eighth grade science, students taught by teachers made significantly larger achievement gains than students taught by Eligible sample teachers in 2012-13, Non-RttT sample teachers in 2013-14, and both comparison groups using two years of pooled data. Examining these overall results by region, the middle and bottom panels of Table G7 indicate that students taught by teachers in the ECU region made larger achievement gains across subjects (mathematics, reading, and science), than students taught by teachers in either comparison group. teachers in the UNC-CSLD region were more effective in mathematics and reading than teachers in either comparison group in the 2012-13 school year but less effective in the 2013-14 school year. teachers in the UNCC region were more effective in mathematics (particularly in 2012-13) and in fifth and eighth grade science; NC NTSP teachers from the UNCG region were less effective than teachers from both comparison groups in mathematics in the 2013-14 school year and less effective than Non-RttT sample teachers in fifth and eighth grade science in 2012-13. Consortium for Educational Research and Evaluation North Carolina 68
The top panel of Table G8 (second page following) shows that for high school grades (nine through twelve) EOC exams in 2013-14 and with two years of pooled data, students taught by evaluation sample teachers made significantly smaller achievement gains than students taught by Non-RttT and Eligible sample teachers. Examining these results by region, the middle and bottom panels of Table G8 indicate that teachers from the UNC-CSLD and UNCG regions were more effective than comparison sample teachers in the 2012-13 school year. teachers in the ECU region were less effective in 2013-14 the coefficients for in the other regions were negative but not significant and with two years of pooled data; teachers in the UNC-CSLD and UNCC regions were less effective than Eligible sample teachers with two years of pooled data. To examine how these value-added estimates may differ by cohort, Tables G9 and G10 (third page following) present results for elementary and middle grades and high school. For the 2012-13 school year, results in the top and bottom panels of Table G9 repeat results from Table G7: Cohort 1 teachers were more effective in mathematics, reading, and fifth and eighth grade science. Examining the 2013-14 school year, where data is available for both Cohort 1 and Cohort 2 teachers, results show that (1) there were no significant differences for Cohort 1 teachers in mathematics or reading; (2) Cohort 2 teachers were less effective in mathematics than Eligible sample teachers; and (3) Cohort 1 and Cohort 2 teachers were more effective than Non-RttT sample teachers in fifth and eighth grade science. With two years of pooled data, Cohort 1 teachers were more effective than both comparison groups in mathematics and fifth and eighth grade science and more effective than Non-RttT sample teachers in reading. For high school EOCs, cohort results in 2012-13 match the overall results reported in Table G8. Results for both cohorts in 2013-14 and with two years of pooled data indicate that Cohort 1 teachers were less effective than Non-RttT sample teachers and that both cohorts were less effective than Eligible sample teachers. Finally, Tables G11 and G12 (fourth page following) present value-added results for first-year teachers only. For the 2012-13 school year, Table G11 shows that students taught by first-year teachers made significantly larger achievement gains in mathematics, reading, and fifth and eighth grade science than students taught by both sets of comparison sample teachers. First-year teachers were also more effective in 2013-14 in fifth and eighth grade science and with two years of pooled data in mathematics and fifth and eighth grade science. High school grades EOC results in Table G12 show that in the 2012-13 school year students taught by first-year teachers made significantly larger achievement gains than students taught by first-year Non-RttT sample teachers; the opposite was true in 2013-14, as first-year teachers were significantly less effective than first-year Non-RttT sample teachers. Overall, evidence from elementary and middle grades indicates that teachers were more effective than comparison sample teachers. These results were concentrated in 2012-13 (Cohort 1 teachers) and did not extend (outside of fifth and eighth grade science) into the 2013-14 school year. Conversely, in the 2013-14 school year, teachers (overall and by cohort) were less effective than comparison group teachers in high school grades EOCs. Consortium for Educational Research and Evaluation North Carolina 69
Table G7. Overall and Regional Value-Added Results (Evaluation Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 vs. Non-RttT 0.194 ** 0.013 0.056 + 0.122 ** 0.000 0.025 0.108 0.128 * 0.122 * vs. Eligible 0.266 ** -0.101 0.087 0.121 * -0.025 0.035 0.189 + 0.102 0.178 * Regions vs Non-RttT ECU Region 0.289 ** 0.086 + 0.125 ** 0.176 ** 0.084 ** 0.111 ** 0.256 + 0.210 + 0.198 + UNC-CSLD Region 0.197 * -0.051-0.006 0.166 ** -0.068 + -0.000 0.076 0.082 0.073 UNCC Region 0.206 ** 0.064 0.111 ** 0.084 + -0.026-0.007 0.113 0.199 * 0.189 * UNCG Region -0.058-0.096 + -0.089-0.038-0.030-0.033-0.193 * -0.040-0.072 Regions vs Eligible ECU Region 0.353 ** -0.034 0.151 * 0.172 * 0.064 0.123 * 0.333 * 0.190 0.256 * UNC-CSLD Region 0.261 ** -0.170 * 0.020 0.162 ** -0.088 + 0.011 0.152 0.063 0.131 UNCC Region 0.269 ** -0.056 0.138 * 0.080-0.046 0.004 0.190 0.179 0.247 * UNCG Region 0.006-0.215 * -0.063-0.042-0.050-0.021-0.116-0.059-0.014 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 70
Table G8. Overall and Regional Value-Added Results (Evaluation Sample) 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams vs. Non-RttT 0.068-0.099 * -0.057 + vs. Eligible 0.019-0.151 * -0.100 ** Regions vs Non-RttT ECU Region 0.045-0.264 ** -0.147 ** UNC-CSLD Region 0.218 + -0.064-0.058 UNCC Region -0.027-0.077-0.066 UNCG Region 0.254 ** -0.067 0.025 Regions vs Eligible ECU Region -0.012-0.292 ** -0.180 ** UNC-CSLD Region 0.161-0.092-0.091 * UNCC Region -0.084-0.105-0.099 + UNCG Region 0.197 * -0.095-0.008 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 71
Table G9. Cohort Value-Added Results (Evaluation Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 Cohorts vs Non-RttT Cohort 1 0.194 ** 0.045 0.102 ** 0.122 ** 0.031 0.064 ** 0.108 0.123 * 0.131 * Cohort 2 --- -0.014-0.004 --- -0.018-0.022 --- 0.132 + 0.106 Cohorts vs Eligible Cohort 1 0.265 ** -0.070 0.122 * 0.121 * 0.007 0.069 0.189 + 0.097 0.185 * Cohort 2 --- -0.130 + 0.016 --- -0.043-0.017 --- 0.106 0.160 + Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G10. Cohort Value-Added Results (Evaluation Sample) Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams Cohorts vs Non-RttT Cohort 1 0.068-0.118 * -0.064 + Cohort 2 --- -0.079-0.044 Cohorts vs Eligible Cohort 1 0.019-0.172 * -0.109 ** Cohort 2 --- -0.133 + -0.089 + Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 72
Table G11. First-Year Value-Added Results (Evaluation Sample) Group vs. Non-RttT vs. Eligible Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 0.205 ** 0.006 0.099 ** 0.088 * -0.017 0.014 0.271 ** 0.202 * 0.192 ** 0.330 ** -0.128 0.176 * 0.118 * 0.038 0.076 0.235 * 0.757 ** 0.286 * Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G12. First-Year Value-Added Results (Evaluation Sample) Group 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams vs. Non-RttT 0.176 ** -0.181 + 0.014 vs. Eligible 0.037-0.180-0.074 Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 73
Student-Level Value-Added Results: Complete and Amended Evaluation Samples Table G13. Overall and Regional Value-Added Results (Complete Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 vs. Non-RttT 0.049 0.014 0.026 0.094 ** -0.004 0.020-0.029 0.118 * 0.062 vs. Eligible 0.159 ** -0.108 0.051 0.085 + -0.029 0.030 0.082 0.056 0.104 Regions vs Non-RttT ECU Region 0.159 * 0.073 + 0.096 * 0.119 ** 0.078 ** 0.099 ** -0.022 0.159 0.096 UNC-CSLD Region 0.050-0.016-0.001 0.132 ** -0.072 ** -0.006 0.005 0.078 0.028 UNCC Region 0.144 * 0.033 0.063 + 0.073-0.024-0.015 0.076 0.202 ** 0.147 + UNCG Region -0.187 ** -0.073-0.113 * -0.049-0.011-0.018-0.314 ** -0.043-0.130 * Regions vs Eligible ECU Region 0.252 ** -0.054 0.115 + 0.108 + 0.057 0.109 * 0.086 0.095 0.144 UNC-CSLD Region 0.143 * -0.144 + 0.017 0.121 * -0.092 * 0.004 0.113 0.014 0.076 UNCC Region 0.237 ** -0.095 0.082 0.062-0.044-0.005 0.184 0.138 0.195 + UNCG Region -0.094-0.201 * -0.094-0.060-0.031-0.008-0.205 + -0.098-0.082 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 74
Table G14. Overall and Regional Value-Added Results (Amended Evaluation Sample) Group vs. Non-RttT vs. Eligible Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 0.189 ** 0.017 0.051 + 0.121 ** -0.003 0.021 0.093 0.117 * 0.107 * 0.260 ** -0.090 0.084 0.119 * -0.025 0.030 0.174 + 0.079 0.162 * Regions vs Non-RttT ECU Region 0.288 ** 0.066 0.109 ** 0.176 ** 0.082 ** 0.105 ** 0.238 + 0.178 0.182 + UNC-CSLD Region 0.182 * -0.006 0.014 0.164 ** -0.070 ** -0.009 0.033 0.048 0.029 UNCC Region 0.204 ** 0.049 0.091 * 0.083 + -0.026-0.007 0.114 0.220 ** 0.191 * UNCG Region -0.058-0.075-0.082-0.037-0.015-0.022-0.219 * -0.037-0.075 Regions vs Eligible ECU Region 0.351 ** -0.046 0.138 * 0.172 * 0.063 0.115 * 0.320 * 0.139 0.245 * UNC-CSLD Region 0.245 ** -0.119 0.042 0.160 ** -0.089 + 0.001 0.115 0.009 0.092 UNCC Region 0.267 ** -0.063 0.119 + 0.079-0.044 0.003 0.196 0.181 0.254 * UNCG Region 0.005-0.187 * -0.054-0.041-0.034-0.012-0.137-0.076-0.012 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 75
Table G15. Overall and Regional Value-Added Results (Complete Sample) 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams vs. Non-RttT -0.025-0.085 * -0.069 + vs. Eligible -0.024-0.096 + -0.079 ** Regions vs Non-RttT ECU Region -0.053-0.311 ** -0.202 ** UNC-CSLD Region 0.081-0.070-0.047 UNCC Region -0.122 * -0.071-0.077 + UNCG Region 0.090-0.061-0.008 Regions vs Eligible ECU Region -0.048-0.308 ** -0.205 ** UNC-CSLD Region 0.085-0.067-0.050 UNCC Region -0.118-0.068-0.080 + UNCG Region 0.094-0.059-0.012 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 76
Table G16. Overall and Regional Value-Added Results (Amended Evaluation Sample) 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams vs. Non-RttT 0.079 + -0.060-0.018 vs. Eligible 0.014-0.083-0.059 + Regions vs Non-RttT ECU Region 0.055-0.292 ** -0.170 ** UNC-CSLD Region 0.174-0.047-0.025 UNCC Region -0.030-0.041-0.011 UNCG Region 0.232 ** -0.043 0.053 Regions vs Eligible ECU Region -0.005-0.300 ** -0.196 ** UNC-CSLD Region 0.114-0.056-0.051 UNCC Region -0.090-0.050-0.037 UNCG Region 0.172 * -0.052 0.027 Note: The top panel of this table displays differences in adjusted-average student achievement between teachers and Non-RttT and Eligible sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between teachers from each region and the reference group of Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 77
Table G17. Cohort Value-Added Results (Complete Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 Cohorts vs Non-RttT Cohort 1 0.049 0.030 0.043 0.094 ** 0.014 0.050 * -0.029 0.168 ** 0.075 Cohort 2 --- 0.004 0.002 --- -0.012-0.013 --- 0.086 0.042 Cohorts vs Eligible Cohort 1 0.159 ** -0.093 0.066 0.085 + -0.011 0.055 0.082 0.105 0.115 Cohort 2 --- -0.119 0.024 --- -0.038-0.007 --- 0.023 0.082 Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G18. Cohort Value-Added Results (Amended Evaluation Sample) Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & Group 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 Cohorts vs Non-RttT Cohort 1 0.189 ** 0.039 0.095 ** 0.121 ** 0.035 0.066 ** 0.093 0.165 ** 0.146 * Cohort 2 --- 0.003 0.006 --- -0.020-0.021 --- 0.088 0.059 Cohorts vs Eligible Cohort 1 0.260 ** -0.067 0.121 * 0.119 * 0.013 0.071 0.174 + 0.122 0.190 * Cohort 2 --- -0.104 0.032 --- -0.042-0.017 --- 0.044 0.103 Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 78
Table G19. Cohort Value-Added Results (Complete Sample) Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams Cohorts vs Non-RttT Cohort 1-0.025-0.102 * -0.084 * Cohort 2 --- -0.076 + -0.054 Cohorts vs Eligible Cohort 1-0.024-0.116-0.095 * Cohort 2 --- -0.091 + -0.065 + Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non-RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G20. Cohort Value-Added Results (Amended Evaluation Sample) Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams Cohorts vs Non-RttT Cohort 1 0.079 + -0.090 + -0.038 Cohort 2 --- -0.042 0.004 Cohorts vs Eligible Cohort 1 0.014-0.119-0.083 * Cohort 2 --- -0.071-0.040 Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Non- RttT sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 teachers and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 79
Table G21. First-Year Value-Added Results (Complete Sample) Group vs. Non-RttT vs. Eligible Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 0.094 + 0.018 0.067 0.079 * 0.001 0.024 0.079 0.156 * 0.131 * 0.265 ** -0.145 0.129 * 0.101 + 0.026 0.075 0.167 0.115 0.177 + Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G22. First-Year Value-Added Results (Amended Evaluation Sample) Group vs. Non-RttT vs. Eligible Elementary and Middle Grades Math Elementary and Middle Grades Reading 5 th and 8 th Grade Science 2012-13 & 2012-13 & 2012-13 & 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 2012-13 2013-14 2013-14 0.205 ** 0.007 0.094 ** 0.088 * -0.016 0.011 0.271 ** 0.203 ** 0.198 ** 0.330 ** -0.127 0.170 * 0.118 * 0.033 0.073 0.235 * 0.756 ** 0.288 * Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and NC NTSP Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 80
Table G23. First-Year Value-Added Results (Complete Sample) vs. Non-RttT vs. Eligible 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams -0.009-0.175 * -0.090 + -0.049-0.121 + -0.118 * Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table G24. First-Year Value-Added Results (Amended Evaluation Sample) vs. Non-RttT vs. Eligible 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams 0.146 ** -0.165 * -0.004 0.037-0.073-0.056 Note: This table displays differences in adjusted-average student achievement between first-year teachers and first-year Non-RttT and Eligible sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 81
Appendix H. To What Extent does the Impact Teacher Effectiveness as Measured by the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? For analyses of teacher evaluation ratings, the Evaluation Team presents four sets of results: (1) overall results for the 2012-13 school year, the 2013-14 school year, and the two years combined; (2) regional results (separate results for teachers in the ECU, UNC-CSLD, UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14 school year, and the two years combined; (3) cohort results (separate results for Cohort 1 teachers entering the program in the 2012-13 year and Cohort 2 teachers entering the program in 2013-14) for the 2012-13 school year, the 2013-14 school year, and the two years combined; 20 and (4) firstyear teacher results, which focus on first-year teachers in the 2012-13 school year, the 2013-14 school year, and the two years combined. 21 Taken together, this set of analyses presents evidence regarding the overall effectiveness of the program and how that effectiveness may have differed by region, cohort, and for first-year teachers. These sub-analyses are particularly important given the differences in treatment components across regions and cohorts and the hypothesis of stronger program effects for first-year teachers. Since only a minority of educators teach in tested grade/subjects and many important aspects of teaching will not be fully captured by teachers value-added to student achievement, the Evaluation Team analyzed teachers observation-based evaluation ratings on the North Carolina Professional Teaching Standards (NCPTS). There are five Standards directly assessed by school principals Demonstrating Leadership (Standard 1), Establishing a Respectful Classroom Environment (Standard 2), Content Knowledge (Standard 3), Facilitating Student Learning (Standard 4), and Reflecting on Practice (Standard 5) and for each Standard school principals rate teachers at one of five levels Not Demonstrated, Developing, Proficient, Accomplished, and Distinguished. For these analyses the Evaluation Team estimated ordered logistic regression models where the outcome variable was a teacher s evaluation rating on the one to five scale (where one was Not Demonstrated and five was Distinguished). These models controlled for teacher experience and school characteristics to help isolate the impact of the on teachers ratings, and clustered standard errors at the school level to account for dependence in the data. Results from these models express the odds of evaluation sample teachers receiving higher evaluation ratings than their Non-RttT and Eligible sample peers. Statistically significant odds ratios greater than one indicate higher evaluation ratings; statistically significant odds ratios less than one indicate lower evaluation ratings. In this appendix, the Evaluation Team presents results for the evaluation sample and for (1) the complete sample of teachers served by the and in the comparison groups, and (2) an amended evaluation sample that includes teachers who worked in non-rttt schools still excluding TFA corps members and those entering the in January or later. 20 Cohort 1 teachers (evaluation sample) include first- and second-year teachers. Cohort 2 teachers (evaluation sample) include first-, second-, and third-year teachers. 21 There are a small number of teachers who have zero years of experience in both the 2012-13 and 2013-14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from the first-year teacher sample in 2013-14. Consortium for Educational Research and Evaluation North Carolina 82
Table H1 (following page) displays overall and regional evaluation rating results. The top panel of Table H1 indicates that there are no statistically significant differences in evaluation ratings between teachers and teachers in either comparison group. Despite the presence of a common, statewide teacher evaluation rubric, examining these evaluation rating results by region must be done with caution since evaluation ratings may differ across distinct regions of the state due to factors unassociated with the. Nonetheless, results in 2013-14 and with two years of pooled data show that teachers in the UNCC region have significantly higher evaluation ratings than their Non-RttT sample peers for Standard 1 (Demonstrating Leadership), Standard 4 (Facilitating Student Learning), and Standard 5 (Reflecting on Practice) and significantly higher evaluation ratings than their NC NTSP Eligible sample peers with two years of pooled data for Standard 2 (Establishing a Respectful Classroom Environment), Standard 4 (Facilitating Student Learning), and Standard 5 (Reflecting on Practice). teachers in the remaining regions ECU, UNC-CSLD, and UNCG are generally rated no differently than their comparison sample peers. To determine whether these evaluation rating results differ by cohort, Table H2 (second page following) presents separate evaluation rating results for Cohort 1 (first served by the in 2012-13) and Cohort 2 (first served by the in 2013-14) teachers. In reference to Non-RttT sample teachers, the top panel of Table H2 indicates that in the 2013-14 school year Cohort 1 teachers had significantly higher evaluation ratings for Standard 1, Standard 3, Standard 4, and Standard 5; Cohort 2 teachers had significantly lower evaluation ratings for Standard 2 in 2013-14 and with two years of pooled data. The bottom panel of Table H2 shows that there were no statistically significant differences in evaluation ratings between Eligible sample teachers and teachers in either cohort. Finally, Table H3 (second page following) presents evaluation rating results for first-year teachers only. The odds ratios indicate that first-year teachers are generally rated no differently than their first-year comparison sample peers only one result, comparing and Eligible sample teachers for Standard 5 in 2012-13 is statistically significant. Across models, results in Tables H1-H3 suggest that, outside of teachers in the UNCC region and Cohort 1 teachers in 2013-14, there was generally no difference in evaluation ratings between and comparison sample teachers. Consortium for Educational Research and Evaluation North Carolina 83
Table H1. Overall and Regional Evaluation Rating Results (Evaluation Sample) 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years Group S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 vs. Non-RttT 1.077 1.114 0.998 0.814 0.981 1.097 0.902 1.006 1.079 1.036 1.072 0.960 0.993 0.975 0.996 vs Eligible 1.267 1.381 1.275 1.096 1.384 0.785 0.767 0.842 0.965 0.811 1.023 1.064 1.033 1.001 1.107 Regions vs Non-RttT ECU 0.755 1.372 0.413 0.485 + 0.900 1.018 0.828 0.879 0.820 1.022 0.878 0.921 0.666 0.655 0.941 UNC-CSLD 1.188 0.793 1.408 0.995 1.097 0.809 0.776 0.866 1.018 0.934 0.834 0.719 0.906 0.896 0.886 UNCC 1.563 1.388 1.450 1.169 1.386 1.972 + 1.650 1.713 1.849 + 1.947 + 1.822 + 1.601 1.613 1.588 1.735 + UNCG 0.854 0.878 1.068 0.811 0.658 0.950 0.697 0.820 0.933 0.672 0.928 0.771 0.931 0.913 0.675 Regions vs Eligible ECU 0.989 2.017 0.598 0.662 1.530 0.809 0.783 0.802 0.776 0.881 0.891 1.085 0.736 0.713 1.118 UNC-CSLD 1.557 1.166 2.037 + 1.357 1.864 0.643 0.734 0.789 0.964 0.805 0.846 0.847 1.002 0.975 1.052 UNCC 2.048 2.040 2.099 1.595 2.354 + 1.568 1.561 1.562 1.750 1.678 1.848 1.886 + 1.783 1.728 + 2.061 * UNCG 1.119 1.290 1.546 1.108 1.119 0.755 0.659 0.748 0.883 0.579 0.941 0.908 1.029 0.994 0.803 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for teachers in reference to Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays odds ratios for each region in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for each region in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 84
Table H2. Cohort Evaluation Rating Results (Evaluation Sample) 2012-13 School Year 2013-14 School Year 2012-13 and 2013-14 School Years Cohort S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 Cohorts vs Non-RttT Cohort 1 1.082 1.128 0.997 0.842 0.980 1.722 * 1.388 1.709 + 1.844 * 1.572 + 1.376 1.267 1.323 1.203 1.257 Cohort 2 --- --- --- --- --- 0.845 0.700 + 0.748 0.811 0.801 0.779 0.658 * 0.694 0.752 0.730 Cohorts vs. Eligible Cohort 1 1.364 1.615 1.378 1.106 1.621 1.270 1.228 1.448 1.628 1.265 1.294 1.391 1.359 1.220 1.399 Cohort 2 --- --- --- --- --- 0.623 0.619 0.634 0.715 0.645 0.732 0.722 0.713 0.763 0.812 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table H3. First-Year Evaluation Rating Results (Evaluation Sample) Group BT1 NC NTSP vs Non-RttT BT1 NC NTSP vs NC NTSP Eligible 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 1.175 1.149 1.121 0.961 1.108 0.840 0.892 1.006 1.029 0.823 1.044 1.054 1.070 1.023 0.948 1.338 1.356 1.111 0.830 1.455 + 0.832 0.494 0.576 0.697 0.925 1.166 1.067 0.902 0.753 1.213 Note: This table displays odds ratios for earning higher evaluation ratings for first-year teachers in reference to first-year Non-RttT and Eligible sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 85
Teacher Evaluation Rating Results: Complete and Amended Evaluation Samples Table H4. Overall and Regional Evaluation Rating Results (Complete Sample) 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 vs. Non-RttT 0.835 0.920 0.965 0.849 0.821 1.071 1.067 1.171 1.133 1.031 0.988 1.011 1.091 1.024 0.945 vs. Eligible 1.054 1.208 1.192 1.088 1.285 0.823 0.922 1.044 0.980 0.875 0.941 1.061 1.096 1.002 1.087 Regions vs Non-RttT ECU 0.534 + 0.886 0.416 + 0.483 * 0.733 0.917 0.763 0.825 0.713 0.823 0.751 0.790 0.623 0.594 * 0.766 UNC-CSLD 0.927 0.766 1.672 + 1.157 1.101 0.858 0.931 0.941 0.988 0.967 0.836 0.835 1.094 0.974 0.963 UNCC 1.390 1.444 1.368 1.164 1.263 1.687 2.147 * 2.489 ** 2.238 ** 1.765 * 1.593 1.922 * 2.014 * 1.824 * 1.559 + UNCG 0.778 0.764 0.973 0.889 0.578 * 0.955 0.801 0.967 0.996 0.750 0.884 0.782 0.975 0.943 0.677 + Regions vs Eligible ECU 0.691 1.262 0.544 0.626 1.233 0.766 0.714 0.786 0.660 0.750 0.762 0.893 0.669 0.622 + 0.929 UNC-CSLD 1.200 1.090 2.188 * 1.500 1.853 * 0.717 0.870 0.896 0.914 0.882 0.849 0.945 1.174 1.020 1.167 UNCC 1.799 2.056 1.790 1.509 2.125 1.410 2.007 + 2.370 * 2.070 + 1.609 1.616 2.174 * 2.162 * 1.910 * 1.889 * UNCG 1.008 1.087 1.274 1.152 0.972 0.798 0.749 0.920 0.921 0.684 0.897 0.884 1.046 0.988 0.820 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for teachers in reference to Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays odds ratios for each region in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for each region in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 86
Table H5. Overall and Regional Evaluation Rating Results (Amended Evaluation Sample) Group vs. Non-RttT vs. Eligible 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 1.011 1.031 0.965 0.783 0.927 1.099 1.073 1.218 1.163 1.079 1.047 1.035 1.135 1.031 1.012 1.225 1.336 1.266 1.081 1.356 0.831 0.967 1.096 1.075 0.890 1.002 1.125 1.161 1.048 1.119 Regions vs Non-RttT ECU 0.745 1.338 0.402 0.482 + 0.893 0.901 0.738 0.830 0.734 0.862 0.805 0.819 0.643 0.606 0.823 UNC-CSLD 1.066 0.732 1.363 0.929 1.023 0.850 0.891 0.977 0.997 0.949 0.818 0.777 0.973 0.869 0.887 UNCC 1.563 1.396 1.459 1.177 1.372 1.750 2.187 * 2.752 ** 2.332 ** 1.895 * 1.694 1.936 * 2.250 ** 1.876 ** 1.714 * UNCG 0.763 0.741 0.965 0.774 0.630 1.033 0.848 1.026 1.041 0.815 0.939 0.818 1.032 0.951 0.751 Regions vs Eligible ECU 0.989 2.010 0.594 0.665 1.523 0.740 0.721 0.796 0.727 0.764 0.819 0.974 0.722 0.668 0.980 UNC-CSLD 1.416 1.100 2.011 + 1.281 1.745 0.698 0.871 0.936 0.987 0.840 0.833 0.923 1.092 0.958 1.057 UNCC 2.076 2.096 2.153 + 1.623 2.340 + 1.437 2.137 + 2.639 * 2.308 * 1.679 1.726 2.301 * 2.525 ** 2.067 * 2.042 * UNCG 1.013 1.113 1.424 1.067 1.074 0.848 0.828 0.984 1.030 0.722 0.957 0.971 1.158 1.048 0.895 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for teachers in reference to Non-RttT and NC NTSP Eligible sample teachers. The second panel of this table displays odds ratios for each region in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for each region in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 87
Table H6. Cohort Evaluation Rating Results (Complete Sample) 2012-13 School Year 2013-14 School Year 2012-13 and 2013-14 School Years Cohort S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 Cohorts vs Non-RttT Cohort 1 0.858 0.925 0.965 0.868 0.847 1.464 1.322 1.570 + 1.652 * 1.413 1.112 1.106 1.224 1.125 1.084 Cohort 2 --- --- --- --- --- 0.927 0.968 1.071 0.985 0.885 0.850 0.899 0.966 0.913 0.798 Cohorts vs. Eligible Cohort 1 1.056 1.261 1.237 1.096 1.398 1.151 1.142 1.356 1.398 1.213 1.056 1.160 1.223 1.094 1.243 Cohort 2 --- --- --- --- --- 0.729 0.837 0.925 0.834 0.760 0.807 0.942 0.965 0.889 0.916 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table H7. Cohort Evaluation Rating Results (Amended Evaluation Sample) 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years Cohort S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 Cohorts vs Non-RttT Cohort 1 1.010 1.030 0.965 0.821 0.931 1.493 1.311 1.601 + 1.713 * 1.477 1.226 1.170 1.286 1.144 1.197 Cohort 2 --- --- --- --- --- 0.959 0.981 1.154 1.017 0.934 0.885 0.905 1.033 0.925 0.842 Cohorts vs. Eligible Cohort 1 1.289 1.505 1.361 1.087 1.550 1.151 1.176 1.382 1.548 1.223 1.164 1.284 1.320 1.159 1.333 Cohort 2 --- --- --- --- --- 0.739 0.879 0.996 0.919 0.773 0.839 0.992 1.060 0.937 0.938 Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 88
Table H8. First-Year Teacher Evaluation Rating Results (Complete Sample) Group vs. Non-RttT vs. Eligible 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 0.889 0.937 0.882 0.940 0.813 0.813 0.994 1.155 1.133 0.811 0.866 0.962 1.046 1.049 0.798 1.118 1.167 0.886 0.745 1.328 0.901 0.622 0.826 0.739 1.011 1.021 0.938 0.875 0.724 1.168 Note: This table displays odds ratios for earning higher evaluation ratings for first-year teachers in reference to first-year Non-RttT and NC NTSP Eligible sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table H9. First-Year Teacher Evaluation Rating Results (Amended Evaluation Sample) Group vs. Non-RttT vs. Eligible 2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 1.105 1.066 1.035 0.948 1.030 0.828 1.031 1.201 1.142 0.924 0.955 1.027 1.138 1.053 0.942 1.303 1.315 1.066 0.830 1.412 0.905 0.717 0.916 0.897 1.052 1.109 1.047 0.995 0.810 1.213 Note: This table displays odds ratios for earning higher evaluation ratings for first-year teachers in reference to first-year Non-RttT and NC NTSP Eligible sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 89
Appendix I. To What Extent does the Impact the Retention of Novice Teachers to the Same School, LEA, and the State? For analyses of teacher retention, the Evaluation Team presents four sets of results: (1) overall results for the 2012-13 school year, the 2013-14 school year, and the two years combined; (2) regional results (separate results for teachers in the ECU, UNC-CSLD, UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14 school year, and the two years combined; (3) cohort results (separate results for Cohort 1 teachers entering the program in the 2012-13 school year and Cohort 2 teachers entering the program in 2013-14) for the 2012-13 school year, the 2013-14 school year, and the two years combined; 22 and (4) first-year teacher results, which focus on first-year teachers in the 2012-13 school year, the 2013-14 school year, and the two years combined. 23 Taken together, this set of analyses presents evidence regarding the overall effectiveness of the program and how that effectiveness may have differed by region, cohort, and for first-year teachers. These subanalyses are particularly important given the differences in treatment components across regions and cohorts and the hypothesis of stronger program effects for first-year teachers. To determine whether evaluation sample teachers were more likely to remain in teaching than their Non-RttT and Eligible sample peers, the Evaluation Team estimated the probability that teachers and teachers from each of the comparison groups would return to teaching in the state. Specifically, the Evaluation Team estimated models for three types of retention: (1) returning to any North Carolina public school in the following school year (2013-14 and 2014-15); (2) returning to the same LEA in the following school year; and (3) returning to the same school in the following school year. For these analyses the Evaluation Team estimated logistic regression models, where the outcome variable was a 1 if the teacher returned in the following school year and a 0 if the teacher did not. In these models the Evaluation Team controlled for teacher experience and school characteristics, to help isolate the impact of the on teacher retention, and clustered standard errors at the school level to account for dependence in the data. Post-estimation, the Evaluation Team converted the odds ratios to predicted retention probabilities to facilitate easier interpretation of the results. In this appendix, the Evaluation Team displays results for the evaluation sample and for (1) the complete sample of teachers served by the and in the comparison groups, and (2) an amended evaluation sample that includes teachers who worked in non-rttt schools still excluding TFA corps members and those entering the NC NTSP in January or later. Table I1 (second page following) presents overall and regional retention results. Pooling data over both evaluation years 2013-14 and 2014-15 the top panel of Table I1 shows that teachers were significantly more likely to return to NCPS than their Non- RttT sample peers. Regarding LEA retention, teachers were significantly more likely to return to the same LEA than Eligible teachers in 2013-14 22 Cohort 1 teachers (evaluation sample) include first- and second-year teachers. Cohort 2 teachers (evaluation sample) include first-, second-, and third-year teachers. 23 There are a small number of teachers who have zero years of experience in both the 2012-13 and 2013-14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from the first-year teacher sample in 2013-14. Consortium for Educational Research and Evaluation North Carolina 90
and Non-RttT sample teachers with two years of pooled data. The strongest NC NTSP retention results concerned returning to the same low-performing school teachers were significantly more likely to return to the same school than Non-RttT sample teachers in 2013-14, 2014-15, and in both years, combined, and more likely to return than Eligible sample teachers with two years of pooled data. Examining these results by region reduces statistical power to detect effects and must be done with caution since rates of teacher persistence differ across the distinct regions of the state due to factors unassociated with the. Nonetheless, results show the strongest retention results, particularly school-level retention, for teachers in the ECU region. teachers from the UNC-CSLD, UNCC, and UNCG regions also have significantly higher school retention rates than Non-RttT sample teachers. To understand how these retention results may differ according to the year of entry, Table I2 (second page following) presents separate retention results for Cohort 1 (first served by the in 2012-13) and Cohort 2 (first served by the in 2013-14) teachers. The top panel of Table I2 indicates that Cohort 1 teachers were significantly more likely than their Non-RttT sample peers to return to NCPS, the same LEA, and the same school. Significant retention results for Cohort 2 were limited to school-level retention. The bottom panel of Table I2 displays fewer significant differences between teachers by cohort and Eligible sample teachers. Cohort 1 teachers were more likely to return to the same LEA in the 2013-14 school year and the same low-performing school in 2014-15 and with two years of pooled data; Cohort 2 teachers were more likely to return to the same low-performing school with two years of pooled data. Finally, Table I3 (second page following) displays retention results for first-year teachers only. These results are significant for first-year teachers (in 2012-13) returning in the 2013-14 school year first-year teachers were more likely than Non-RttT sample teachers to return to NCPS and more likely than both comparison groups to return to the same LEA and the same school but there are no significant differences for first-year teachers (in 2013-14) returning in 2014-15. Coupled with the values in Tables I1 and I2, results suggest that the had its strongest retention impacts on teachers returning to the same low-performing school and for Cohort 1 teachers. Consortium for Educational Research and Evaluation North Carolina 91
Table I1. Overall and Regional Retention Results (Evaluation Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 87.66 84.99 85.74 79.61 76.67 77.47 74.11 72.72 73.04 Non-RttT 84.36 82.53 83.09 + 74.37 73.94 73.98 + 67.14 + 64.95 ** 65.75 ** Eligible 86.11 87.65 86.76 72.11 + 76.81 74.49 65.66 66.44 65.99 * Region Results vs. Non-RttT Group ECU Region 90.14 86.31 87.62 + 83.80 79.26 80.49 + 81.39 * 75.25 * 77.26 ** UNC-CSLD 86.16 83.70 83.88 72.71 74.49 73.26 66.21 73.15 * 70.22 Region UNCC Region 86.15 84.02 84.85 77.84 74.66 76.32 72.67 71.71 72.66 + UNCG Region 88.67 86.05 86.99 83.29 78.39 79.91 * 74.89 70.80 72.35 + Non-RttT 84.39 82.54 83.11 74.51 73.91 74.07 67.31 64.91 65.78 Region Results vs. Eligible Group ECU Region 90.14 86.31 87.62 83.80 + 79.26 80.49 81.39 * 75.25 + 77.26 ** UNC-CSLD 86.16 83.70 83.88 72.71 74.49 73.26 66.21 73.15 70.22 Region UNCC Region 86.15 84.02 84.85 77.84 74.66 76.32 72.67 71.71 72.66 UNCG Region 88.67 86.05 86.99 83.29 78.39 79.91 + 74.89 70.80 72.35 Eligible 86.24 87.67 86.77 72.19 77.12 74.51 65.81 66.61 66.02 Note: The top panel of this table displays predicted probabilities of retention for teachers and teachers in each comparison group. The second panel of this table displays results for each region in comparison to the Non-RttT group; the bottom panel of this table displays results for each NC NTSP region in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 92
Table I2. Cohort Retention Results (Evaluation Sample) Group Cohort 1 Cohort 2 Non-RttT Cohort 1 Cohort 2 Eligible Returns in 2013-14 Returns to NCPS Returns to the Same LEA Returns to the Same School Overall: Overall: Overall: Returns in 2013-14 & Returns in Returns in 2013-14 & Returns in Returns in 2013-14 & 2014-15 2014-15 2013-14 2014-15 2014-15 2013-14 2014-15 2014-15 87.66 87.87 ** 86.87 + 79.61 79.59 + 79.08 + 74.11 + 75.76 ** 74.18 ** --- 83.54 84.45 --- 75.28 75.61 --- 71.12 * 71.67 * 84.36 82.48 83.07 74.37 73.82 73.95 67.14 64.92 65.74 87.66 87.87 86.87 79.61 + 79.59 79.08 74.11 75.76 + 74.18 * --- 83.54 84.45 --- 75.28 75.61 --- 71.12 71.67 + 86.11 87.65 86.80 72.11 76.97 74.68 65.66 66.47 66.05 Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table I3. First-Year Teacher Retention Results (Evaluation Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 BT1 87.29 81.79 84.04 80.22 72.82 75.70 77.42 67.99 71.72 Non-RttT 81.03 * 81.81 81.18 71.35 * 74.21 72.42 65.43 ** 66.37 65.58 * Eligible 83.51 86.46 83.87 68.62 * 74.21 70.11 61.04 ** 67.47 63.11 * Note: This table displays predicted probabilities of retention for first-year teachers and first-year teachers in the Non-RttT and Eligible groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 93
Teacher Retention Results: Complete and Amended Evaluation Samples Table I4. Overall and Regional Retention Results (Complete Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 85.46 84.21 84.60 77.27 75.52 76.04 69.66 71.88 71.24 Non-RttT 82.85 81.23 81.72 + 72.56 72.81 72.63 + 65.65 62.71 ** 63.81 ** Eligible 83.41 88.03 85.72 68.87 * 77.23 72.93 62.41 63.61 * 62.99 ** Region Results vs. Non-RttT Group ECU Region 85.67 83.33 84.64 76.56 73.37 74.85 67.53 69.49 + 69.37 + UNC-CSLD Region 81.71 82.73 82.01 70.90 74.16 72.72 64.72 72.61 ** 69.74 + UNCC Region 86.02 83.91 84.74 77.56 76.32 76.89 69.71 74.08 * 73.14 ** UNCG Region 88.17 86.97 * 87.26 * 83.37 * 78.28 + 79.91 ** 75.65 * 70.92 + 72.59 ** Non-RttT 82.95 81.31 81.81 72.71 72.86 72.72 65.81 62.71 63.88 Region Results vs. Eligible Group ECU Region 85.67 83.33 84.64 76.56 73.37 74.85 67.53 69.49 69.37 + UNC-CSLD 81.71 82.73 82.01 70.90 74.16 72.72 64.72 72.61 + 69.74 + Region UNCC Region 86.02 83.91 84.74 77.56 + 76.32 76.89 69.71 74.08 + 73.14 * UNCG Region 88.17 86.97 87.26 83.37 ** 78.28 79.91 * 75.65 * 70.92 72.59 * Eligible 83.14 87.94 85.53 68.50 77.06 72.66 62.11 63.39 62.72 Note: The top panel of this table displays predicted probabilities of retention for teachers and teachers in each comparison group. The second panel of this table displays results for each region in comparison to the Non-RttT group; the bottom panel of this table displays results for each NC NTSP region in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 94
Table I5. Overall and Regional Retention Results (Amended Evaluation Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 86.95 84.99 85.48 78.71 76.46 77.08 72.82 72.93 72.97 Non-RttT 84.21 82.37 82.91 + 74.23 73.74 73.75 + 67.12 64.83 ** 65.57 ** Eligible 86.02 87.55 86.78 72.09 76.57 74.33 65.62 66.43 + 66.02 * Region Results vs. Non-RttT Group ECU Region 90.04 85.09 86.80 83.73 76.20 78.29 81.25 * 72.64 * 75.16 * UNC-CSLD Region 84.03 84.35 83.70 70.36 74.75 73.05 63.70 73.52 ** 70.25 UNCC Region 85.78 84.05 84.75 77.30 76.60 77.39 72.14 73.58 + 74.01 * UNCG Region 88.01 86.55 87.05 + 81.95 78.49 79.63 * 72.84 71.89 72.67 * Non-RttT 84.28 82.39 82.95 74.41 73.75 73.87 67.31 64.82 65.62 Region Results vs. Eligible Group ECU Region 90.04 85.09 86.80 83.73 + 76.20 78.29 81.25 * 72.64 75.16 * UNC-CSLD 84.03 84.35 83.70 70.36 74.75 73.05 63.70 73.52 + 70.25 Region UNCC Region 85.78 84.05 84.75 77.30 76.60 77.39 72.14 73.58 74.01 + UNCG Region 88.01 86.55 87.05 81.95 78.49 79.63 + 72.84 71.89 72.67 Eligible 86.07 87.56 86.75 72.06 76.58 74.15 65.70 66.40 65.87 Note: The top panel of this table displays predicted probabilities of retention for teachers and teachers in each comparison group. The second panel of this table displays results for each region in comparison to the Non-RttT group; the bottom panel of this table displays results for each NC NTSP region in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 95
Table I6. Cohort Retention Results (Complete Sample) Group Cohort 1 Cohort 2 Non-RttT Cohort 1 Cohort 2 Eligible Returns in 2013-14 Returns to NCPS Returns to the Same LEA Returns to the Same School Overall: Overall: Overall: Returns in 2013-14 & Returns in Returns in 2013-14 & Returns in Returns in 2013-14 & 2014-15 2014-15 2013-14 2014-15 2014-15 2013-14 2014-15 2014-15 85.46 86.51 + 85.05 + 77.27 76.45 76.59 69.66 73.63 ** 70.72 ** --- 83.29 84.08 --- 75.11 75.42 --- 71.18 ** 71.82 ** 82.85 81.22 81.72 72.56 72.82 72.62 65.65 62.70 63.81 85.46 86.51 85.05 77.27 * 76.45 76.59 69.66 73.63 * 70.72 * --- 83.29 84.08 --- 75.11 75.42 --- 71.18 + 71.82 ** 83.41 88.03 85.74 68.87 77.21 73.01 62.41 63.63 62.95 Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 96
Table I7. Cohort Retention Results (Amended Evaluation Sample) Group Cohort 1 Cohort 2 Non-RttT Cohort 1 Cohort 2 Eligible Returns in 2013-14 Returns to NCPS Returns to the Same LEA Returns to the Same School Overall: Overall: Overall: Returns in 2013-14 & Returns in Returns in 2013-14 & Returns in Returns in 2013-14 & 2014-15 2014-15 2013-14 2014-15 2014-15 2013-14 2014-15 2014-15 86.95 87.76 * 86.30 + 78.71 77.91 77.80 72.82 74.78 * 73.11 ** --- 83.86 84.71 --- 75.90 76.41 --- 72.15 ** 72.82 ** 84.21 82.32 82.90 74.23 73.66 73.75 67.12 64.82 65.57 86.95 87.76 86.30 78.71 77.91 77.80 72.82 74.78 + 73.11 + --- 83.86 84.71 --- 75.90 76.41 --- 72.15 72.82 * 86.02 87.51 86.81 72.09 76.63 74.44 65.62 66.43 66.02 Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 teachers in comparison to the Non-RttT group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 teachers in comparison to the Eligible group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Table I8. First-Year Teacher Retention Results (Complete Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 BT1 86.28 81.37 83.55 78.65 71.94 74.84 72.11 67.49 69.54 Non-RttT 81.92 + 82.54 82.00 71.63 * 74.41 72.77 65.83 + 65.26 65.35 Eligible 80.63 + 86.70 82.24 65.08 ** 75.56 68.11 + 57.43 ** 63.82 59.15 ** Note: This table displays predicted probabilities of retention for first-year teachers and first-year teachers in the Non-RttT and Eligible groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 97
Table I9. First-Year Teacher Retention Results (Amended Evaluation Sample) Returns to NCPS Returns to the Same LEA Returns to the Same School Group Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15 BT1 87.16 82.07 84.00 79.58 72.90 75.36 76.62 68.24 71.35 Non-RttT 80.80 * 81.79 81.07 71.11 * 74.29 72.34 65.17 ** 66.51 65.50 * Eligible 83.41 85.69 83.75 68.64 * 73.10 69.88 60.98 ** 66.84 63.07 * Note: This table displays predicted probabilities of retention for first-year teachers and first-year teachers in the Non-RttT and Eligible groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively. Consortium for Educational Research and Evaluation North Carolina 98
Contact Information: Please direct all inquiries to Kevin Bastian kbastian@email.unc.edu 2015 Consortium for Educational Research and Evaluation North Carolina