Transcription

1 1 of 6 10/25/2018, 10:52 AM UNIT REPORT Intensive English Program Assessment Report 2018 Generated: 10/25/18, 10:51 AM International Education Intensive English Program Assessment Plan Describe Annual Assessment Plans: Our Assessment Committee continues to meet twice during the semester. Some changes were made in individual classes regarding assessment. The grammar classes had been using the on-line textbook support for exams, but we found that too many items were multiple choice or fill-in-the-blank. We added more items that required grammar production rather than recognition. Production of the structure is a better gauge of students' mastery. In writing classes we continue to focus on in-class writing rather than out-of-class assignments. This ensures that the students are doing their own writing and are able to work within a time restraint--something that will be required of them in academic classes. The instructors in the speaking classes are effectively using technology to enhance their classes. In the advanced speaking class students prepared individual videos that gave advice to future new international students. The students learned to edit their videos. Finally, they were compiled and presented as a capstone project. We continue to assess student performance as a class. We measure overall class performance against each stated student learning outcome. Performance is rated as "Developing," "Competent," or "Superior." The performance on the SLOs are analyzed to see which ones need more attention. The Assessment Committee works on strategies to improve performance in those areas of instruction. As a staff, we have decided to continue with our assessment tools that we used in We will continue to use the rubrics that we have created. In terms of specific changes, we have discussed several changes regarding specific classes. In the beginning speaking and understanding, we are going to create a better instrument for assessing listening skills. We do not have a reliable test that is designed for listening students at the beginning level. At the intermediate level, we will introduce basic instruction with comparative and superlative forms. Rather than doing a specific unit, however, the instructors will incorporate them as they teach various verb tenses. Instructors will assess both skills at the same time. In intermediate speaking and understanding the instructors will spend more time assessing speaking. In writing, the intermediate students will do all of their graded writing in class to prevent students getting help from electronic devices. At the advanced level, the speaking and understanding classes will focus more on listening. Students currently do a lot of oral presentations, but we feel there is a shortage of listening activities. Instructors are going to have students do more note taking and more completions of rubrics as they listen to their classmates make oral presentations. As in intermediate writing, the advanced writers will do all of their graded writing in class. At the end of the 2016 spring semester IEP faculty used the curriculum maps and rubrics tool in Campus Labs to assess individual classes. The basic guidelines for IEP instructors were as follows: For each class, determine where the majority of that class ranked for each language learning outcome on the curriculum map. Based on how the majority of the class accomplished the stated learning outcome, the class was ranked as Developing, Competent, or Advanced. The faculty used copies of the curriculum maps as a quick and easy checkoff method for their determination. After compiling the results, we met as a staff to analyze the data. At this time we did two things. First, we reviewed the learning outcomes for each class to determine if we wanted to reword existing outcomes or add new ones. Second, we analyzed the results of the spring semester s learning outcomes and deliberated how we might improve upon classroom performance. At this time we brainstormed and made determinations about how to maintain or improve the results that were reported. Before the start of the fall semester, we will meet again to reassess the curriculum maps and discuss what we can do to improve on the results reported from the spring semester. To meet requirements from our accrediting agency, CEA (Commission on English Language Program Accreditation), an assessment committee will meet twice each semester to review our curriculum Accreditation), an assessment committee will meet twice each semester to review our curriculum maps and assessment practices and how we might improve assessment and reporting of assessment in the future. The Intensive English Program at Emporia State University is accredited by the Commission on English Language Program Accreditation

2 2 of 6 10/25/2018, 10:52 AM (CEA). The program aligns with the Standards set forth by the commission and assessment practices are implemented to insure compliance with CEA standards. During the spring semester, we formulated a plan to articulate the basic standards for each course at each level beginning, intermediate, and advanced. During the summer, we will work on how to evaluate those standards. Our basic plan is to measure the class for each standard we have chosen. We will determine where the majority of the class scores at each standard" developing," "competent," or "advanced." Our goal is to get 80% of the class to the "advanced" rating for each standard. Start: 07/01/2015 End: 06/30/2025 Department Summary, Strategies, and Next Steps: The IEP staff continues to assess the overall competency of a class rather than tracking individual assessments. For each class, the instructors use a rubric with the student learning outcomes listed. Regarding the overall performance of the class, the instructors check one of three categories: Developing, Competent, or Superior. A designation of Developing indicates no evidence of skill for the SLO. Competent indicates minimal competency for the indicated SLO. Superior indicates mastery of the skill has been demonstrated by the majority of the students in the class. We continue to work as a staff on the best method for student assessment for the program. Speaking and Understanding In the speaking and understanding classes, we began focusing on oral interviews in the first few days of the semester. In the past, large student numbers hindered us from doing oral interviews as part of the placement procedure. We focused primarily on listening. However, we decided that an early oral interview would be helpful, especially to target students early in the semester who might benefit with some one-on-one oral practice in our tutoring lab. Writing Assessments in writing classes focused primarily on composition topics for program-wide assessments: placement writing exam, diagnostic writing exam, and final writing exam. For the placement and diagnostic exams, we tried to avoid topics with complex rhetorical modes. Instead, we focused on wording topics carefully so that the topics dealt mainly with enumeration or narrative and topics that could be understood by writers at all skill levels. ReadingThe reading classes continue to pose a problem of finding ways to assess students reading skills with authentic materials. By the end of the semester, the reader will have been limited to just a few topics from the textbook. To this end, we have collected a wide variety of reading passages that we can draw from. This gives us a better idea of whether the student has mastered the skills that were covered in class. Our main challenge in the grammar classes concerns finding a good balance of assessment materials. Most of the current textbooks come with a test bank that instructors can load into the computer. Unfortunately, most of the questions on these exams are based on recognition rather than production. Most of the questions from the test bank are multiple choice. In order to strengthen our assessment procedures and to comply with our accreditation agency, CEA, we established an assessment committee that meets twice each semester--at the beginning and at the end. As a committee, we decided to use our rubric templates in the spring to assess our classes. Each class was evaluated in terms of overall success. How did the majority of the class do in meeting the standards for each learning outcome? Faculty used a check-off system and rated the class as Developing, Competent, or Superior. Our goal was to have the majority of the class at the Superior level. If that did not happen, the committee strategized about improving class performance. We will continue to evaluate both the learning outcomes listed and the overall proficiency of each class. As next year develops we will initiate the use of the rubrics developed to measure standards as matching with the CEA accreditation requirements. CEA Standards for Accreditation - IEP Program 2015 Program Name : Reading and Speaking Programs Summary of Program Assessments: In classes that teach reading, we find that students can often find the answer to a question from a reading, but the answer provided is usually

3 3 of 6 10/25/2018, 10:52 AM straight from the text being read. We are implementing a new strategy that will combine assessments of both reading and speaking skills. In both intermediate and advanced reading classes, when a student is asked to provide an answer from a written text, the student will try and articulate a response using his/her own words rather than relying on the text and just repeating words directly from the text. This will integrate both reading and speaking skills. The student will employ the skill of scanning to find the answer in the text. The student will use the context to help clarify meaning. Finally, the student will think about what the text says and try to put the answer into different words. Even with just a slight variation from the text, the student is employing a form of paraphrasing which is a high-level skill. We continue to use our rubrics to gauge the success of classes on our student learning outcomes. Our assessment committee continues to discuss what student learning outcomes to focus on. Current rubrics evidence is found in the file library. In summary of Reading assessments we seek to determine if students can demonstrate proficiency from a reading text with which they are not familiar? We continue to try and incorporate critical reading skills, but realize as second-language learners that students often lack the vocabulary sophistication to adequately express themselves with this higher level concept. Our rubric assessment guide has proven to be valuable in determining if the reading skills are being learned as expected. With respect to the Speaking program, instructors doing the oral interviews have a master list of speaking prompts that they can draw from. Each student is rated on answers to five questions. Instructors use a rubric to rate the students on their overall performance.in terms of listening, we have been relying heavily on Randall s ESL Listening Lab via the internet. It provides short graded conversations at three levels: Easy, Medium, and Difficult, so they can be used at all three levels of our program. The conversations are followed with 5 multiple choice questions. They provide instructors with a quick assessment tool that can be done in the classroom. The rubrics evidence files are located in the file library. The rubrics were completed in the summer of 2015 and the curriculum map was completed this past 2016 academic year. Over the course of the year, faculty engaged in employing a rubric to assess the reading and speaking courses being taught. The data and findings from the oneyear assessment cycle serve as a baseline for comparison data for the 2017 academic year and subsequent years. As mentioned, the faculty committee has reviewed student performance results and are planning strategies to improve student learning. Using the data from the first year, individual faculty are making course level and assignment level changes, but strategies at the program levels will not be determined until multiple iterations of data can inform decisions. Rubrics results are shown in the evidence files for beginning, intermediate, and advanced courses. IE-002-Intermediate-Speaking--Understanding-2017 IE-004-Intermediate-Reading-2017 IE-006-Advanced-Speaking--Understanding-2017 IE-008-Advanced-Reading-2017 IE-012-Beginning-Speaking--Understanding-2017 IE-014-Beginning-Reading-2017 LAS-IE-ENGLISH SPEAKING-FA15 LAS-IE-ENGLISH READING-FA15 IE-002-Intermediate-Speaking--Understanding IE-004-Intermediate-Reading IE-006-Advanced-Speaking--Understanding IE-008-Advanced-Reading IE-012-Beginning-Speaking--Understanding IE-014-Beginning-Reading IE-002-Intermediate-Speaking--Understanding-2018.pdf IE-004-Intermediate-Reading-2018.pdf IE-006-Advanced-Speaking--Understanding-2018.pdf IE-008-Advanced-Reading-2018.pdf Program Name: Grammar and Writing Programs Summary of Program Assessments: Our assessment committee looked at student learning outcomes in writing and grammar and looked at the specific problem of the disconnect

4 4 of 6 10/25/2018, 10:52 AM between grammar class learning and writing class production. We wanted to see how we could see more evidence of writers using the grammar concepts that students had learned in the grammar class. We want to see more writing in the grammar classes as part of the assessment tool. Instead of having students recognize the grammar by filling in a blank or choosing from multiple answers, we have the students use the structure by writing sentences or short responses that require the grammar concept. Our exams are focusing more and more on production. Results show that students are meeting student learning outcomes at a greater rate as indicated in our reporting in the file library. For the grammar component, as a staff, we have agreed on the need to supplement book provided tests with questions that deal with production. Students should be able to demonstrate that they can produce the grammar concept that has been covered in class. Grammar instructors try to work up test elements that require students to write sentences that demonstrate their mastery of the targeted grammar concept. Another assessment tool that has been effective is to have students present an oral activity such as a skit or narrative that requires the use of the grammar element in question. For the writing final exam essay, we expanded topic choices to include rhetorical modes that the students had studied during the semester. Overall, we felt that we were now providing topics that were more suited to students skills at different stages of assessment. Assessment findings (2017 rubrics scores in the file library) showed that most of the students are progressing as they should. The results of the rubrics scores for 2017 are shown in the file library. Similar to the reading and speaking program, the rubrics and curriculum maps were used to apply assessment techniques and scoring student work determined the student success levels across beginning, intermediate, and advanced level courses. The data will be used to inform program level decisions when data accumulated over multiple semesters can be compared. The results did show that the majority of students are proceeding through the program successfully. It is noted that depending on the given characteristics of the students entering the program, different weaknesses and needs emerge. We are beginning the conversations about how we can bring a more in-depth approach to understanding the data as it applies to different learner groups. This is true based on where the students are originally from (native country), and also the language which is considered native. It is always a challenge to adapt to these incoming student populations to ensure that learning is occurring as it should to prepare them for success in a major program of study. Rubrics evidence files are shown in the file library. IE-001-Intermediate-Structure-2017 IE-003-Intermediate-Writing-2017 IE-005-Advanced-Structure-2017 IE-009-Advanced-Writing-2017 IE-011-Beginning-Structure-2017 IE-013-Beginning-Writing-2017 LAS-IE-ENGLISH WRITING-FA15 LAS-IE-ENGLISH GRAMMAR-FA15 IE-001-Intermediate-Structure IE-003-Intermediate-Writing IE-005-Advanced-Structure IE-009-Advanced-Writing IE-011-Beginning-Structure IE-013-Beginning-Writing IE-001-Intermediate-Structure-2018.pdf IE-003-Intermediate-Writing-2018.pdf IE-005-Advanced-Structure-2018.pdf IE-009-Advanced-Writing-2018.pdf Progress: Ongoing over Multiple Years Feedback on Assessments: The progress being made in changing the performance measures to incorporate higher order learning skills is a very nice change to the existing pedagogy and curriculum. Incorporating technology into the learning experience advances students abilities to use technology as a

5 5 of 6 10/25/2018, 10:52 AM learning tool and can be used as students advance to study in their major discipline, as well. The integration of the learning experiences such as combining the learning of grammar with the product of writing, and the understanding of reading with the product of explaining through speaking should produce higher levels of learning competencies in all areas. Changing the measuring instruments to include more original works instead of identification and selection in tests will also positively improve the student learning experience and competencies. It will be important to have faculty dialogue about the potential impact of the rubric scoring on the newer mechanisms for measuring student learning. If necessary, you may need to have a rubric norming session where a few faculty look at common assignments and then score them to the rubrics, to verify that faculty are interpreting student performance in similar ways. I say this because the faculty are used to scoring on the rubrics with previous assignments and may not make universal adjustments to the new assignment types. It may be worthy of a conversation. If you have questions about this rubric norming concept, please reach out to me. The faculty working together to assess the students' learning and collaborating to improve the student learning experience is a strength of the program. Overall, your faculty dedicate their efforts to continuously improving the student learning experience and incorporate key strategic changes on an annual basis. This is a best practice. Maintaining these concerted assessment efforts among the faculty is encouraged, as there are many positive benefits for our students educational experiences. Thank you for all you do! The faculty of the IE Program have continued to work together to make changes across the curriculum at all of the levels of learning for the key content areas of the program including Reading, Speaking, Grammar, and Writing. In reviewing the results of the rubrics scores and progress for this past 2017 academic year, most students are learning at levels which are preparing them to matriculate into a university level curriculum. You have expanded your beginning of the year placement evaluations and this appears to be a key information piece in determining where some students are struggling with the curriculum and possibly identifying where the resource materials being used for instruction are falling short. You have made some nice changes based on this information and having faculty engage in dialogue to improve the student learning experience is a key assessment strategy that is always encouraged. As with having new students entering into the IE program every year, you will have year-to-year fluctuations in knowledge and skill sets based on the country of origin and whether English is a second, third, or fourth language. The faculty have a quality system in place to identify the nuances of these first-time students and it is encouraging to see that curricular changes are being made annually to adjust to these changing student demographics. You will want to begin preparations for the program reaccreditation in the upcoming year. Keep up the good work and continue to engage faculty in dialogue relating to student learning improvement strategies. Nice Job! The Intensive English Program has made good progress with designing the assessment infrastructure to inform its program. The curriculum maps and rubrics have served to collect a years worth of student learning performances and are setting a baseline to inform curricular and pedagogical decisions. A suggestion for next year is to include the beginning of term placement evaluations. These data can serve to inform curriculum decisions and improve the ability to make adaptations to serve student needs prior to the beginning of classes (keeping faculty abreast of anticipated challenges). This information can also be used to show the progress students have made from the time they begin the program until they complete it. You can also look at term-to-term progress, as well. This information can be very valuable for faculty as it shows the areas where students are excelling in their learning, while also identifying areas of need. This type of data will also help identify the differences among learner groups that has served as challenges in the past. This is good assessment work, continue to engage your faculty in conversations about how student learning can be maximized, this is key to improving student success across all three learning levels. The alignment with the assessment plan with the CEA specialized accreditation standards is a best practice. Student successes on these standards measured by rubrics will enable you to look at student performances (formative assessments) on a term by term basis and will also allow for some trend information (summative assessments) to also drive your decision-making, curriculum changes, and pedagogical strategies. Although this assessment program is newly developed, the set up is solid and should serve you well in the future. Continue emphasis on the curriculum mapping piece where you define and refine student learning outcomes at the course level, then align the courses with the program student learning outcomes. You have this in place already, but the refinement will occur as you adjust to external accreditation requirements. Also, let the findings from your assessments drive your future directions. Pick those areas where the students are having struggles or not quite learning as the faculty would like. Some of the challenges the IEP faces may be linked to the constant changing of the student population you serve. Adapting to these challenges will be served well by the assessment plan. You might want to look at the feedback loops the students have to share their thoughts. Learning about how students conceptualize their learning can help identify learning strategies which are strengths for them. This may help in the short term adaptations and strategies. In addition, the rubrics results should provide some direction in next steps as well. Be sure to include them in next year's report. Providing Department: International Education - Intensive English Program