1. Continuous Assessment Results a. Admission Data English for Secondary Teachers Initial Preparation Annual Program Report Academic Year 2012 13 Alex Poole November 14, 2013 English for Secondary Teachers 201213 Page 1 of 14 Table 1 provides the average admission test scores and admission grade point average (GPA) of English for Secondary Teachers candidates approved by the Professional Education Council (PEC) for admission into initial teacher preparation programs during this academic year. Before the Office of Teacher Services submits their names for review and approval by the PEC, candidates must meet minimum requirements established by the state and/or the WKU Professional Education Unit. Table 1. Approved Candidate Test Score Averages PPST PPST PPST GRE Admission ACT SAT Program Math Reading Writing Composite GPA N Mean N Mean N Mean N Mean N Mean N Mean N Mean English for Secondary Teachers 28 25 30 3.17 b. Course Based Assessment Data Table 2 provides the percentage of English for Secondary Teachers candidates (N = 84) scoring at each level of proficiency on critical performances within education courses for this academic year. Proficiency levels are based on a scale of 1 Standard Not Met, 2 Standard Partially Met, 3 At Standard, and 4 Above Standard. Table 2. CP Proficiency Level Percentages Course 1 2 3 4 ED 201 0 0 100 0 EDU 250 0 2.56 74.36 23.08 EDU 489 0 16.67 83.33 0 ELED 345 0 0 33.33 66.67 ELED 355 0 0 50 50 ELED 365 0 0 100 0 LME 448 0 0 100 0 LTCY 320 0 0 0 100 LTCY 421 0 0 100 0 MGE 275 0 0 0 100 PE 121 0 0 100 0 PSY 310 0 0 21.88 78.13
Course 1 2 3 4 SEC 351 0 49.69 50.31 0 SEC 352 0 48.08 51.92 0 SEC 453 0 0 21.33 78.67 SEC 475 0 0 100 0 SPED 330 0 0 0 100 Grand Total 0 3.30 55.88 40.82 English for Secondary Teachers 201213 Page 2 of 14 Table 3 indicates the level of English for Secondary Teachers candidates (N = 84) proficiency across critical performances related to the Kentucky Teacher Standards (KTS). Candidates receiving an overall rating of 3 or 4 on a CP are considered to have demonstrated proficiency on the standards associated with the CP. Compared to the unit wide results, English for Secondary Teachers candidates are typically performing average. Table 3. Percent of English for Secondary Teachers Candidates Scoring Proficient on CPs by KTS Program Kentucky Teacher Standards 1 2 3 4 5 6 7 8 9 10 EST 96.93 97.56 96.33 100 94.94 96.7 96.43 94.55 97.41 100 Unit Wide 97% 97% 97% 98% 97% 97% 97% 98% 97% 99% *KTS Key: 1 Content Knowledge, 2 Designs/Plans Instruction, 3 Maintains Learning Climate, 4 Implements/ Manages Instruction, 5 Assessment/Evaluation, 6 Technology, 7 Reflection, 8 Collaboration, 9 Professional Development, 10 Leadership Table 4 indicates the number of English for Secondary Teachers candidates (N = 4) who have scored 2 or lower (below proficiency) on critical performances during this academic year. If there are no 800#s indicated in the table provided, then your program had zero students who scored below proficient. Table 4. English for Secondary Teachers Candidates Scoring Below Proficient on CPs Student ID Score 1 2 Student Count 800725200 1 1 800480325 1 1 800706129 2 2 c. Clinical Experiences Data Grand Total 4 4
Page 3 of 14 The EST program uses the following courses and experiences to evaluate candidate dispositions: EDU 250 and EDU 490. The program has identified the following courses and experiences where candidates report the diversity of their field experiences: EDU 250, EDU 480, and SEC 352. SEC 352 has been designated as the experience where candidates must work in settings at or above the average 11% diversity of the schools in the 30+ counties that represent our service area. Table 5 reports how English for Secondary Teachers candidates performed on dispositions as they entered and progressed through their program (N =74) and during their student teaching experience (N = 17). Students are considered proficient who average a 3 or higher on each disposition category. Table 5. English for Secondary Teachers Proficiency Rates on Unit Wide Dispositions WKU Professional Education Dispositions Period Values Learning Values Personal Integrity Values Diversity Values Collaboration Values Professionalism a. Prior to Student Teaching 92.22 100 100 98.15 100 b. During Student Teaching 100 100 100 100 100 Over this academic year, English for Secondary Teachers candidates (N = 63) reported demographic information on 115 field placements with an average of 16% ethnically diverse students, 46.4% students on free/reduced lunch, and 10.2% student with disabilities based on National Center for Education Statistics and Kentucky Department of Education). Table 6 reveals the percentages of field experiences with various characteristics. Note that candidates could choose all the characteristics that applied for any given experience. Table 6. Percentages of Field Experience by Category Types Working with Student With Special Needs % Candidates working with Students with Physical Impairments 9 % Candidates working with Students with Learning Disabilities 48 % Candidates working with Students with Moderate/Severe Disabilities 10 % Candidates working with Students with Emotional/Behavioral Disorders 30 % Candidates working with Gifted Students 68 % Candidates working with English Language Learners 33 % Candidates working with Students with Visual Impairments 0 % Candidates working with Students with Hearing Impairments 4 % Candidates working with Students with Speech/Language Delays 8 % Candidates working with Students with Development Delays 5 % Candidates working with Students with Autism Spectrum Disorder 11 % Candidates working with Students with Other Impairments 1
Page 4 of 14 Working with Diverse Students % Candidates working with African American Students 91 % Candidates working with Native American/American Indian Students 12 % Candidates working with Latino/Hispanic Students 79 % Candidates working with Asian Students 56 % Candidates working with Students with Special Needs (Aggregate) 88 % Candidates working with Diverse Students (Aggregate) 94 Overall, as can be seen in Table 6, in 88% of their field experiences, EST candidates reported working with at least one student with special needs and in 94% of their field experiences candidates reported working with at least one student from a diverse ethnic group d. Culminating Assessment Data As Component 4 of the WKU Professional Education Unit Continuous Assessment Plan (CAP) strategy, all initial preparation candidates complete a culminating assessment of professional and pedagogical knowledge and skills, the Teacher Work Sample (TWS). This assessment is also used to demonstrate candidates ability to impact P 12 student learning. In particular, candidate performances on Assessment Planning and Analysis of Student Learning have been identified as key indicators of candidates ability related to student learning. Although in spring 2008 the Professional Education Council agreed that candidates who score a holistic score of at least 2 Developing are able to exit the program, for program evaluation purposes our goal is that at least 80% of program candidates will achieve 3 Proficient or higher. Table 7 presents the proficiency rate for EST candidates (N = 17). Table 7. Initial Preparation TWS Proficiency Rates Program % Proficient EST 91.84 Unit Wide 93% Because the faculty also scores TWS at the indicator level, we are able to use these scores to ascertain candidate success in meeting each component of the TWS. For program evaluation purposes, candidates are considered successful who average at least 2.5 on a three point scale (1 Not Met, 2 Partially Met, and 3 Met) on indicators aligned to a standard. Table 8 depicts the percentage of EST candidates who averaged at least 2.5 on the indicators for each TWS Factor: CF Contextual Factors, LG Learning Goals, DFI Design for Instruction, ASL Analysis of Student Learning, and ROT Reflection on Teaching. Table 8. Initial Preparation TWS Proficiency Rates of English for Secondary Teachers Candidates Program CF LG DFI ASL ROT EST 94.12 82.35 70.59 94.12 100
Unit Wide 89% 91% 85% 91% 88% English for Secondary Teachers 201213 Page 5 of 14 Because the TWS indicators have been aligned to Kentucky Teacher Standards, we can use these scores to ascertain candidate success in meeting each standard related to the TWS. Table 9 reports these scores as they relate to Kentucky Teacher Standards. Table 9. Percentage of English for Secondary Teachers Candidates who Passed each Teacher Standard Program 1 2 3 5 6 7 9 EST 82.35 94.12 94.12 82.35 88.24 100 100 Unit Wide 89% 94% 92% 90% 92% 91% 88% Additionally, all candidates are assessed during their student teaching experience using the Student Teaching Evaluation form. Table 10 reports the percentages of English for Secondary Teachers student teachers (N = 17) successful on each standard. For program evaluation purposes, candidates are considered successful who average at least 2.5 on a three point scale (1 Not Met, 2 Partially Met, and 3 Met) on indicators aligned to a standard. Table 10. English for Secondary Teachers Proficiency Rates by Kentucky Teacher Standards Program Kentucky Teacher Standards 1 2 3 4 5 6 7 8 9 10 EST 100 94 100 88 88 88 94 94 94 94 Unit Wide 97% 92% 96% 91% 87% 90% 86% 93% 97% 89% e. Exit and Follow Up Data Table 11 delineates the Educational Testing Services reports of the pass rates on the Praxis II content exams of candidates who completed the program in the 2011 12 academic year (the most recent year with complete data). The last column allows for pass rate comparison of our candidates to our 2010 11 results. Table 11. Pass Rates on Content Tests for Initial Teacher Preparation Program/Type of Assessment Candidate N (2011 12) WKU Pass Rate (2011 12) WKU Pass Rate (2010 11) ENG LANG LIT COMP CONTENT KNOW 11 100% 96% ENG LANG LIT COMP ESSAYS 10 80% 96% Annually, the WKU Teacher Survey is sent to student teachers and alumni who potentially have been teaching one or more years. For the 2012 13 academic year, out of a possible 17 student teachers, 16 (94%) completed the survey. Below are the results for English for Secondary Teachers student teachers, 16 of whom responded. Survey items requested the respondent s perception of WKU preparation on each of the Kentucky Teacher Standards using a scale of 1
Page 6 of 14 Poor, 2 Fair, 3 Good, and 4 Excellent. Standards with average scores of 3 or better across items were considered to demonstrate acceptable program quality. Table 12 reports English for Secondary Teachers survey results. Table 12. Average Scores on Teacher Standards Questions for English for Secondary Teachers Respondents Program Kentucky Teacher Standards 1 2 3 4 5 6 7 8 9 10 EST 3.31 3.00 3.45 3.09 2.59 3.28 2.85 2.59 3.30 3.11 Unit Wide 3.49 3.45 3.66 3.43 3.29 3.37 3.30 3.14 3.36 3.18 Respondents were also able to provide comments. Table 13 presents English for Secondary Teachers respondent comments. Table 13. EST Respondent Comments I think the biggest weakness in the program was preparing future teachers with effective differentiation techniques. There was little to no instruction in this in the introductory teacher courses. The SEC 352 course, designed to address diversity in the classroom, was VERY poorly designed and taught. I'm not sure if this was an issue with the professor or the program. I felt (and continue to feel) very unprepared to differentiate for diverse learners ranging from ESL to gifted to struggling readers. In addition, I thought there was a large disconnect between theory and practice in several of the teacher education courses. If I answered "poor" to any of the questions, I feel like these areas were only covered during my student teaching semester. I did not feel prepared to deal with these situations when I was going to be assessed over them after only just hearing about them. I met the requirements, but I think all of the teacher standards should be covered before student teaching. Overall, I do not believe that WKU prepared me for student teaching. I feel that certain education classes only taught me the difference in the common core standards. I feel that the only education class that remotely prepared me for student teaching was the one I took with Dr. White (I can't remember if that was SEC 475 or 453). Dr. White actually taught us what the high school atmosphere is like and what teachers can and cannot do. I feel that, instead of teaching and focusing more on the standards or Bloom's taxonomy, our education courses should help us with making more formative and summative assessments. I feel that I was deeply scrutinized by my University Supervisor and that I didn't live up to her standards. It is not fair that each University Supervisor has their own expectations and demands when other supervisors are fairly lax in theirs. The expectations of university supervisors should be more uniformed so that other student teachers can collaborate and don't already feel more defeated and confused. Also, I wish that I would have had more English classes that were geared towards Education majors I don't feel that I was prepared
Page 7 of 14 enough to teach certain novels and short stories because I didn't read or study them in college. "Standard 1d: More sections could have been given by the English department on various types of literature (for example, Asian) so that the student could utilize those texts in the classroom when he or she begins teaching. Standard 2e: There is so much of an emphasis on ""higher level thinking"" in education classes. Of course, this is what a teacher should strive for, but education classes should also emphasize practical strategies that could be implemented to teach lower level students basic skills that must be mastered before higher level thinking can effectively be employed in a classroom setting (how are teachers supposed to get students to analyze the motivation of a character in a text if the students can't even comprehend what a text is saying?)." Student teaching needs to be filled with University Supervisors that want to be hands on and understanding about what happens during the time as a student teacher. My university supervisor was little to no help to me and often made me feel like I was incapable or stupidfor the lack of a better term. I am still lost when it comes to data analysis, so I would definitely suggest incorporating that into classes prior to student teaching. I think it is impossible to analyze data without knowing the data of the students who are being analyzed, but some practice would have been helpful. Dr. Stobaugh did the best she could with the limited amount of time that she has with us. She was very attentive and answered questions/ gave feedback promptly. She gave me more confidence than my Western Supervisor that I met with regularly. The only assessment or critical performance in which the teacher candidates worked with analyzing data wasn't until this year during student teaching with the TWS. I think it should have been implemented earlier on in our education career. "The teaching program at WKU needs to be seriously revised. I did not feel prepared at all for student teaching. If I was to identify how WKU could have better prepared me for each category that I rated ""Poor"" it would be a novel. Instead of asking us to observe for a million hours we need to practice teaching many classes for an extended amount of time. Instead of going into schools and watching others teach, the student teachers need to develop lessons, create assessments, and feel what it is like to be an actual teacher. I had almost no preparation for creating assessments (quizzes, tests, essay prompts, etc.) and no preparation in analyzing assessments to see where I needed to improve. Also, I strongly feel that the TWS and the student teaching portfolio need to be completed at separate times. During student teaching we are not only preparing lessons each day, grading papers, attending extra curricular events (for the leadership project), and working on the collaborative project, but we are also expected to complete the teacher work sample
Page 8 of 14 and our portfolio at the same time. I got very sick during student teaching because of all the stress, lack of sleep, and lack of nutrition and I had to make up the days I missed at the end of the placement abroad. No wonder I got sick! I never slept and barely had time to eat! I don't know what the solution is, but I know that the program needs to better prepare student teachers for the pressure they will be under while student teaching." When it comes to teaching to a variety of students' needs, the education was lacking. I feel that we were given several bits of information such as when you have a student who is ESL do this. Or when the student has this IEP do this. But it was more of a topic that was only discussed for one class period, if that. We are required to take a class that is supposed to teach us to plan for diversity, but it didn't actually teach me how to plan lessons around children with special needs or other disabilities. We also aren't taught how to give tests and accurately assess them. I can remember being told how to make tests and the pros/cons of different types of tests in PSY 350, but not how to assess them in the manner we are expected in the TWS. WKU did a thorough job of preparing me for teaching. It's hard to fully be aware of what teaching will be like before you are actually in the classroom, but WKU did well preparing me for teaching through studying theory and best practice. WKU needs more input from teachers that are still teaching in the school system. I have been told multiple times that the things I have been taught are not relevant. I even had students questioning the "cute and fun" assignments that was drilled into my head that I have to do to make lessons relevant. I also believe that there should be more of an overview and expectations given to students before student teaching. The English Department's Quality Enhancement Plan (QEP) includes three "Intended Educational (Student) Outcomes" which are examined as students near the completion of their initial preparation program. Below are the outcomes, means of assessment and criteria (where available), and most recent results. SPECIFIC GOALS: 1. Students will demonstrate the ability to critically analyze a literary text (NOTE: literary can be defined broadly to include film, pop culture, and other texts that are formally studied in English courses). 2. Students will show the ability to use secondary sources to support an argument about a literary text. 3. Students (self selected) will understand the process and protocol of applying to graduate/professional school OR students (self selected) will understand the process and protocol of applying for a job related to their field of study.
Page 9 of 14 TARGET: For all three goals = 75% score 4 or higher, aggregate, none below 3 RESULTS (9 Literature Majors, 8 EST Majors assessed through portfolios generated in ENG 492): Lit Major Goal One: Critical Analysis of Text 89% (8/9) scored 4 or higher, none below 3 Goal Two: Use of Sources 67% (6/9) scored 4 or higher; none below 3 Goal Three: Career/Grad School 89% (8/9) scored 4 or higher, none below 3 EST Majors Goal One: Critical Analysis of Texts 87% (7/8) scored 4 or higher; none below 3 Goal Two: Use of Sources 75% (6/8) scored 4 or higher; none below 3 Goal Three: Career/Grad School 87% (7/8) scored 4 or higher; 1/15 below 3
Page 10 of 14 Narrative/Closing the Loop Background: The assessment committee made two significant changes to the program assessment process in 2012 13. 1) Only students who submitted portfolios in the spring semester were assessed. In past years we have assessed every portfolio. This reduced the total number of portfolios evaluated by individual faculty from 17 to 8. 2) The portfolios were made available electronically, meaning that faculty could assess on their own time rather than during one block of time. Both of these changes were designed to improve our assessment results by giving faculty more time to spend with fewer total portfolios. Goal One continues to be strong. We have met or surpassed our target the past three years, meaning that we can retire this goal for now and select another one to assess for 2013 14 The results for Goal Two in 2011 12 were quite disappointing. In response, the department agreed to continue to emphasize the use of sources across the departmental curriculum, with particular targets in ENG 299 and in Survey courses. In 2012 12, the results improved dramatically, from 50% to 67% meeting the target among students in the literature concentration and from 60% to 75% among EST majors. It would seem prudent to assess this again in hopes of seeing even more improvement. Goal three: After years of assessing this, with very mixed results, we have FINALLY met our goals. We probably ought to assess it again, to make sure this was not a fluke. Actions for 2013 14 Select three specific goals to assess for 2013 14 Decide whether to continue the changes made in the assessment process: 50% of portfolios and electronic scoring For some time now it has not been reasonable to call this a program assessment and the program assessment committee, as it only looks at two slices of our department. We need a new name and clear rotation of people on the committee. Should we continue to draw from everyone in the department?
Page 11 of 14 Notes The differences in scoring among individual reviewers were much less than in past years. Only 20% of the collected scores for individual portfolios showed a variation of more than one point from the three reviewers, and none showed a variation of more than two points. Over half of the cases where we had a difference of two points were in the career/grad school category, which suggests we need some additional norming there. One possibility for this improved correlation in scoring is that the reviewers had more time to spend with each portfolio as we switched to the electronic versions. Of course it may just be a one year fluke as well. The list that we use to choose specific things to assess was created over a decade ago. The department may want to update this document. QEP Goals 1) English Majors will be aware of study abroad opportunities. Measurement: Senior exit survey asked students to rate their knowledge of study abroad opportunities on a scale of 1 (lowest) to 5 (highest). Results: 5 = 17 4 = 10 3 = 6 2 = 3 1 = 0 75% 4 or higher 8% Below 3 This is an improvement over last year, especially in the number who strongly agreed. 2) English majors will have studied minority and non Western literature. Measurement: Senior exit survey asked students the following: As an English major, I have formally studied minority and non Western literature Results: Strongly agree = 5 Agree = 18 Neutral = 8 Disagree = 7 Strongly Disagree = 3 55% Agree or Strongly Agree 24% Disagree or Strongly disagree
Page 12 of 14 These numbers dropped precipitously from the previous year. Perhaps it is in part because students are more aware of the presence and importance of this literature. Even though the survey did not ask for comments, many wrote in things like only in World Lit, Not enough!!!, just touched base not enough depth, too many courses ignore it.
Page 13 of 14 Summary of Results by Kentucky Teacher Standards and Other Key Conceptual Framework Values a. Admission Data: The EST ACT mean (25) was slightly higher than the mean of all teaching candidates (24). However, the EST GPA mean (3.17) was lower than the unit wide mean(3.36). Although we would like our students to have higher GPAs, we believe that part of this difference can be attributed to rigorous departmental demands and the fact that we are one of the largest units (N=30), and thus we naturally have a wider range of candidates. At least onethird of the other programs have fewer than ten students. b. Course Based Assessment Data: CP proficiency level percentages ranged from 94.55 to 100. Four were higher than the unit wide average, while four were lower; however, such differences were not numerically robust. c. Clinical Experiences Data: The EST proficiency rates on the Unit Wide Dispositions differ little prior to and during student teaching; however, in two categories, our students improved from 92.22 and 98.15 to 100 for both. These categories were values learning and values professionalism. These data indicate that are students grow during student teaching. d. Culminating Assessment Data: Figures for the Teacher Work Sample show excellent performance by our students. Overall, our proficient rate was slightly below the unit mean (91.84 vs. 93). Specifically, they scored higher than the unit mean in three categories and below it in two. Of special concern is DFI, which was far below the unit means (70.59 vs. 85). The EST proficiency rates for all Kentucky Teacher Standards are also good, with seven being higher than the unit mean. e. Exit and Follow up Data: EST students continue to improve their performance on Praxis II exams. On the "Content Knowledge" test, 2010 2011 students had a 100% pass rate. On the "Essays" test, the pass rate was 80%. Last year s pass rates for these tests were 96% for both; thus, while one score went up, the other one dropped. However, these results are based on very small number (Content Knowledge=11; Essays=10), and thus can be strongly affected by outliers. Efforts to Report and Disseminate Results: This report was forwarded to the English Department's English for Secondary Teachers Committee for reading and discussion. At the beginning of the semester, Praxis and English capstone course results were distributed to and discussed by the department as a whole. Key Discussions and/or Decisions Made Based on Assessment Results
Page 14 of 14 a. Assessment or Data Collection Changes Based on Assessment Results. We will continue to differentiate the EST majors from all other English majors. By doing this, we will get a more accurate picture of the former s accomplishments and needs. It may also help us determine whether or not unique assessment goals need to be created for EST majors. However, as noted above, we made significant changes last year to our departmental assessment in terms of data collection. It is likely that we will change one of our assessment pieces (critical analysis of text) next year. b. Program Curriculum or Experiences Changes Based on Assessment Results None. c. Decisions about Group/Individual Student Progress Based on Assessment Results No significant changes have been made. As reported above, we made significant changes to our curriculum five years ago. Also noted above is that we made significant changes to our departmental assessment last year in terms of data collection, and it is likely that we will change one of our assessment pieces (critical analysis of text) next year. Discuss trends in assessment results over the last few years (Please refer back to your 2009 10 and 2010 11 APRs which are posted to the College of Education Professional Education Unit website) Our students continue to generally perform well on all assessments. Of course, there are fluctuations on certain specific assessments from year to year, yet given the relatively small numbers of students in each assessment cycle, this is statistically probable.