Assessment for Russian Studies Program Academic Year Submitted by E. A. Blake (Coordinator) on 5/31/2017

Transcription

1 I. Program Assessment Annual Reporting II. Core Assessment A. Program Assessment Plan B. Summary of Data From Assessment III. B. A. Assessment A. Program Assessment Plan B. Summary of Data From Assessment Assessment for Russian Studies Program Academic Year Submitted by E. A. Blake (Coordinator) on 5/31/2017

2 Saint Louis University Program Assessment Annual Reporting It is recommended program assessment results be used to celebrate achievements of student learning as well as to identify potential areas for future curriculum improvement. Please this completed form as an attachment to 1. Degree Program(s) included in this report: Russian Major; Core Requirement in the Russian Language 2. Department: Department of Languages, Literatures, and Cultures 3. School/Center/College: College of Arts and Sciences 4. Name(s): Elizabeth Blake and Yelena Belyaeva Phone: (314) Instructions: Please answer the following five questions to the best of your ability for each degree program offered within your department. 1. Summarize your assessment activities during the past year for each degree program and how this work relates to the established assessment plan (e.g. what program outcomes were assessed, faculty discussions, new survey design, data collection, revised assessment plans or learning outcomes, etc.). Please include how Madrid courses/program were involved. 1. Assessment for the major was developed by the program coordinator based on ACTFL guidelines, a review of similar programs, and experience with modes and guidelines discussed at a Fulbright-Hays orientation. Implementation began this year, but the six majors/1 minor (2 in the Fall; 3 in the spring; 1 in June) graduating were not all assessed, since only three of them took the 4XXX-level course this year in which the first assessment was conducted. 2. Assessment for core language proficiency (Interpersonal communication) was conducted as a part of the oral interviews at the end of Raw data was collected on each individual student and subsequently summarized in prose for the purposes of meaningful reporting. 3. Assessment for the major was introduced at the 4XXX-level in the form of a final project assessing presentational modes in speaking and writing. Data was collected and summarized for three students in prose.

3 2. Describe specific assessment findings related to the learning outcomes assessed for each degree program, including any pertinent context surrounding the findings. Please include the learning outcomes themselves. Do not include student-level data. Data included in this report should be in aggregate. Please include how Madrid courses/program were involved. BA Assessment Presentational Communication (Speaking) for two: -Presentational (Speaking): Student can make a presentation in a generally organized way on school, on education, and on researched topics. Presentational Communication (Writing) for three: -Presentational (Writing): Student can write on topics relating to school, work, community, and researched topics (requiring interpretive reading and/or listening) in a generally organized way in simple paragraphs with various time frames represented. As far as assessment for the major was concerned, out of the three majors assessed this year at the 4XXX level, two exceeded expectations and one failed to meet expectations. Native knowledge and study abroad opportunities allowed two out of three to exceed expectations of the Intermediate-Mid range in writing (based on assessment of the presentational mode writing and speaking) while two out of two exceeded expectations in speaking. Core Assessment Interpersonal Communication The Core Assessment was conducted in December 2016 with little advance notice as to the nature of the assessment, so the only skill assessed was Interpersonal Communication (Speaking) as part of the oral interview at the end of Of the nine who were assessed, five out of the nine exceeded expectations, three met expectations, and one of nine did not meet expectations. Realizing that arriving at the expectations for the core level did not translate into preparedness for the courses constituting the major at the 3XXX level, the Russian program added the 2020 requirement for the major courses in the spring. *Please attach any tables, graphics, or charts to the end of this report. 3. Describe how assessment feedback has been provided to students, faculty, and staff. (e.g. report for faculty, executive summary for the dean, web page for students, alumni newsletter, discussion with students in class or club event, etc.) Individual feedback was discussed with the majors, but as this is the first year for

4 assessment, we are awaiting better aggregate data for more meaningful comparisons. The assessment reporting for will be forwarded to Kathleen Thatcher. 4. In what ways have you used assessment findings to celebrate student achievements and/or to improve the curriculum this past year? (e.g. prizes to students, hosting student parties, changes to curriculum, student projects, learning goals, assessment strategies, etc.) 1. Finding that students were not meeting personal goals for language development, new textbooks were introduced this year that provide more opportunities for conversation and interactive learning. 2. A new course was added this year to the curriculum RUSS 2020 to improve the quality of minors/majors, and it will be required for incoming freshmen this year. 3. On-line exercises for students having finished the first year and wanting some practice over the summer are in development to support language maintenance and intercultural knowledge outside of the classroom. 4. A student participated in the senior symposium and the departmental symposium with the project, on whose basis he was assessed. (A second student was prevented from participating in the departmental symposium by military duties.) This was posted to the DLLC Facebook page. 5. Another indirect measure of student success is the honors society Dobro Slovo, to which five out of six of our graduating majors belonged in There was a small celebration this year. 6. One of our students from RUSS 2010 received a grant to participate in the STARTALK program at California State University (Northridge) for the summer, and this achievement was posted to the College of Arts and Sciences page. 7. Multiple students were recognized on the departmental achievement board. An ATLAS presentation, as well as a student lecture at the beginning of the year were also ways to celebrate student study abroad experiences (Bard-Smolny). 8. Another indirect measure is that a 2016 graduate from the Russian Studies program will be starting an MA program in Slavic Languages and Literatures at University of Kansas in the fall. 5. Describe any changes to your assessment plans, or any challenges or educational experiences with the assessment process this past year that you would like to share. 1. Assessment for the Russian program was developed and implemented by the part-time tenured and the tenure-track faculty members, but it is a goal for the program that it be integrated more uniformly and intentionally. This will be easier now given that the assessment plan is in place before the beginning of the academic year. However, with a new full-time staff person entering the program, it is expected that

5 assessment will still be a challenge for the next year. Communication among the program's faculty members regarding expectations and implementation will be essential to the success of assessment for the coming year. Better records will be kept of communication within the program regarding assessment responsibilities for each faculty member. 2. For continuity's sake, the faculty members involved in the development of the assessment goals in the year will be teaching the courses (2010 and 4010) in which Core and B.A. assessment are conducted. The current guidelines have been primarily the work of the current program coordinator, who was not kept fully informed about previous developments and so began work on assessment in B. A. assessment is being introduced as part of the 4XXX-level courses, so it is expected that these courses will undergo further development. Since the assessment process was not outlined until late in the fall semester, there is room for improvement in the processes and in the program's being intentional about assessment. Please submit any revised/updated assessment plans to the University Assessment Coordinator along with this report.

6 II. A. Saint Louis University Program Assessment Plan Program Assessment (Majors, Non-Majors: LLC Core Program - LLC 1010, 1020, 2010) Department: Language, Literatures & Cultures College/School: Arts & Sciences Person(s) Responsible for Implementing the Plan: LLC Core Language Program Coordinator Date Submitted: Program Learning Outcomes Curriculum Mapping Assessment Methods Use of Assessment Data What do you expect all Where is the outcome How do students demonstrate How does the program use students who complete learned/assessed (courses, their performance of the program assessment results to the program to know, or be able to do? internships, student teaching, clinical, etc.)? learning outcomes? How does the program measure student performance? Distinguish your direct measures from indirect measures. recognize success and "close the loop" to inform additional program improvement? How/when is this data shared, and with whom? Interpersonal Communication : Students will be able to interact in the target language spoken and written exchanges to share information, reactions, feelings and opinions. LLC 2010 Students will be assessed by the ACTFL proficiency guidelines at the level determined by each program. Example Direct Measures: 1 Summative composition 2 Oral proficiency evaluations 1 Skit or dramatization Example Indirect Measures: Self-assessment ACTFL I-can questionnaires Exit Survey The Program Coordinator or 2010 level-coordinators compiles the results of both the direct and indirect measures across the program in order to gain insights into the gains made by students, the success of instruction and needs for improvement. Based on the Core program report, the LLC Program Coordinator reports to the LLC program at the end of each academic year with recommendations for changes or improvements to the curriculum, implementation and assessment plan.

7 Interpretive Communication: Students will be able to understand and respond to what is heard, read or viewed in the target language on a variety of topics related to themselves and their immediate context. LLC 2010 Students will be assessed by the ACTFL proficiency guidelines at the level determined by each program. Example Direct Measures: 1 Summative composition 1 Writing sample 1 Writing section of the final exam Example Indirect Measures: Self-assessment ACTFL I-can questionnaires Exit survey The Program Coordinator or 2010 level-coordinators compiles the results of both the direct and indirect measures across the program in order to gain insights into the gains made by students, the success of instruction and needs for improvement. Based on the Core program report, the LLC Program Coordinator reports to the LLC program at the end of each academic year with recommendations for changes or improvements to the curriculum, implementation and assessment plan. Presentational Communication: Students will be able to present information, concepts and ideas in the target language to inform, explain, narrate and persuade on a variety of topics using appropriate media and adapting to various audiences. LLC 2010 Students will be assessed by the ACTFL proficiency guidelines at the level determined by each program. Example Direct Measures: 1 Final project and formal presentation 1 IPA project presentation 1 Skit or dramatization Example Indirect Measures: Self-assessment ACTFL I-can questionnaires Exit survey The Program Coordinator or 2010 level-coordinators compiles the results of both the direct and indirect measures across the program in order to gain insights into the gains made by students, the success of instruction and needs for improvement. Based on the Core program report, the LLC Program Coordinator reports to the LLC program at the end of each academic year with recommendations for changes or improvements to the curriculum, implementation and assessment plan.

8 Intercultural Competence: Students will be able to use the target language to investigate, explain and reflect on the nature of language through crosslinguistic and crosscultural comparisons. LLC 2010 Example Direct Measures: 1 Culture comparison reflection paper 1 Culture comparison task as part of the final exam 1 Culture project through immersion experience Example Indirect Measures: Self-assessment intercultural awareness and goals questionnaires Exit survey The Program Coordinator or 2010 level-coordinators are in regular contact with faculty and staff providing extracurricular cultural events and community partners who provide opportunities for community engagement and cultural experiences to our students. Any recommendations for improvements to the culture and service component of the course are reported at the end of the academic year to the Department Chair and the LLC program. The Program Coordinator or 2010 level-coordinators compiles the results of both the direct and indirect measures across the program in order to gain insights into the gains made by students, the success of instruction and needs for improvement. Based on the Core program report, the LLC Program Coordinator reports to the LLC program at the end of each academic year with recommendations for changes or improvements to the curriculum, implementation and assessment plan

9 II. B. Summary of Data From Assessment Russian Studies, Core Inasmuch as the guidelines suggest that not every outcome should be assessed on an annual basis and as the core assessment guidelines were shared late in the semester, the program chose a single measure of assessing interpersonal communication in an oral interview at the end of the semester. Interpersonal Communication: Students will be able to interact in the target language spoken and written exchanges to share information, reactions, feelings and opinions. 56% (5/9) exceeded expectations, 33% (3/9) met expectations, and 11% (1/9) did not meet expectations. This class was unusually strong by comparison with previous years (probably because out of the more than twenty with which we began in 1010, only six remained) and had three students who had transferred into the program, one of which had substantial native knowledge of the language. Five students exceeded the Novice-High level, three students met this level, and one did not meet the expectations for this discourse, based on an interpersonal exchange of predictable questions as discussed in the ACTFL guidelines for the Novice-High/Intermediate-Low levels. Most had some grammatical errors expected of first-year students with inaccurate usage of cases, conjugations, and memorized word combinations, but those that exceeded expectations introduced unexpectedly complex information or word combinations in subjects on personal, family, and student life. Repeated failure to understand basic questions on these topics (even when re-phrasing and repetition were employed), the usage of English, and the presence of incomprehensible word combinations were errors that distinguished the student who did not meet expectations

10 Summary of Data From Assessment Russian Studies, B. A. II. B. Saint Louis University Program Assessment Plan Program (Major, Minor, Core): Russian (Major) Department: Languages, Literatures, and Cultures College/School: Arts & Sciences Person(s) Responsible for Implementing the Plan: Department Chair Date Submitted: November 1, 2016 Program Learning Outcomes What do you expect all students who complete the program to know, or be able to do? -Student will be able to communicate with confidence in Russian in interpersonal, interpretive, and presentational modes at the Intermediate- Mid level (based on ACTFL Can-Do statements). -Interpersonal: Student can handle short social interactions in Curriculum Mapping Assessment Methods Assessment Data Where is the outcome learned/assessed (courses, internships, student teaching, clinical, etc.)? 1. Interpersonal communication: An oral exit interview after the completion of a RUSS 4XXX language course. 2. Presentational communication (Speaking): This should be assessed when the student presents as part of the final requirements for a RUSS 4XXX How do students demonstrate their performance of the program learning outcomes? How does the program measure student performance? Distinguish your direct measures from indirect measures. Direct measure: 1. Interpersonal communication: Oral interview 2. Presentational communication (Speaking): Presentation within the context of a RUSS 4XXX language course. 3. Presentational How does the program use assessment results to recognize success and "close the loop" to inform additional program improvement? How/when is this data shared, and with whom? The departmental assessment committee will review assessment results and will share them with full-time faculty in the Russian division. Changes to the program will be made in consultation with the Chair.

11 everyday situations by asking and answering a variety of questions. -Presentational (Speaking): Student can make a presentation in a generally organized way on school, on education, and on researched topics. language course. 3. Presentational communication (Writing): This should be assessed at the end of RUSS 4XXX Senior Inquiry, Senior Residency or Capstone course. communication (Writing): Brief composition in simple paragraphs to be included as part of the student's portfolio. -Presentational (Writing): Student can write on topics relating to school, work, community, and researched topics (requiring interpretive reading and/or listening) in a generally organized way in simple paragraphs with various time frames represented. - Student can demonstrate will be able to articulate distinctive characteristics of Russian culture in the fields of architecture, art, history, literature, music, philosophy, political science, and theology. - Undergraduate majors will demonstrate an ability to analyze critically famous Assessment: -In the student's portfolio will be kept assignments from courses that attest to this intercultural competency. -This can be assessed in connection with courses RUSS 2XXX, RUSS 3XXX, RUSS 4XXX. Direct measure: Oral interview or presentation in connection with RUSS 3XXX, RUSS 4XXX, or Senior Inquiry. Indirect measure: -Participation in study abroad. -Placement of graduating seniors into related graduate and professional programs. The departmental assessment committee will review assessment results and will share them with full-time faculty in the Russian division. Changes to the program will be made in consultation with the Chair.

12 narratives in prose and will articulate how Russian literary, historical, and cultural traditions are situated within global contexts. 1. It is not recommended to try and assess (in depth) all of the program learning outcomes every semester. It is best practice to plan out when each outcome will be assessed and focus on 1 or 2 each semester/academic year. Describe the responsibilities, timeline, and the process for implementing this assessment plan. Each division in LLC will assess outcomes annually, beginning in the academic year. In the Russian program, assessment for the core takes place in the fall semester whereas assessment for the major will occur in the spring semester. Each major will have a file, into which selected assignments attesting to the successful completion of learning outcomes will be placed. It will be the responsibility of the fulltime faculty members in the program to complete and monitor assessment. 2. Please explain how these assessment efforts are coordinated with Madrid (courses and/or program)? This is not coordinated with Madrid. 3. The program assessment plan should be developed and approved by all faculty in the department. In addition, the program assessment plan should be developed to include student input and external sources (e.g., national standards, advisory boards, employers, alumni, etc.). Describe the process through which your academic unit created this assessment plan. Include the following: a. Timeline regarding when or how often this plan will be reviewed and revised. (This could be aligned with program review.) The plan will be reviewed every 7 years: the year before the department s program review. b. How students were included in the process and/or how student input was gathered and incorporated into the assessment plan. Students were not involved in the development of this plan. c. What external sources were consulted in the development of this assessment plan? ACTFL: American Council on the Teaching of Foreign Languages The College of Wooster's Curricular Review ( )

13 III. B. Summary of Data From Assessment Russian Studies, B. A. Inasmuch as the guidelines suggest that not every outcome should be assessed on an annual basis and as the program is phasing in assessment, three out of the six graduating majors/ one minor were assessed as part of the 4XXX-level course. The primary emphasis was on writing, but speaking was assessed for two out of three of the students. 1. Presentational (Speaking): Student can make a presentation in a generally organized way on school, on education, and on researched topics. 100% (2/2 of those assessed) exceeded expectations. On their researched topics, both students exceeded outlined expectations, since they spoke for an extended period on their topics for which they had conducted web and print research in the target language. Their ability to synthesize knowledge drawn from different resources (written and video) in the target language, the presence of advanced-level phrases in their speech, and their accurate usage of the target language speak to their development toward professional-level discourse and therefore exceed the expectations for the program, partly because of knowledge of Russian gained outside of the program. 2. Presentational (Writing): Student can write on topics relating to school, work, community, and researched topics (requiring interpretive reading and/or listening) in a generally organized way in simple paragraphs with various time frames represented. 66% (2/3 of those assessed) exceeded expectations while 34% (1/3 of those assessed) did not meet expectations. The written assessment of the three students who were in the 4XXX-level course for the major differed to some extent, because one faculty member designed the assignments being assessed for two students while the third completed a slightly different assignment for assessment. The two who exceeded expectations did benefit from knowledge of Russian gained outside of the program. They demonstrated an ability to understand and integrate into their writing professional-level terminology and to write a comprehensive and coherent summary of their research. The one student who did not meet expectations did write in paragraph-length discourse that was sometimes comprehended by one familiar to speaking with non-natives, but the student failed to demonstrate a consistent knowledge of even first-year grammar structures such as verbnoun agreement, case usage (with verbs and prepositions), adjective-noun agreement, and the learning verbs. This suggests that in a spontaneous discussion in Russian, a native speaker would fail to comprehend his sentence-length discourse. This would suggest that the first year needs to continue to be a focus of development for the strength of the program.