2017 Assessment Leadership Summer Institute

Transcription

1 2017 Assessment Leadership Summer Institute Evaluation Report May 15, 2018 Assessment Office Crawford Hall 230 & Campus Road, Honolulu, HI (808) (808) manoa.hawaii.edu/assessment

2 Assessment Leadership Summer Institute 2017 Assessment Leadership Summer Institute Evaluation Report Assessment Office 05/15/2018 Table of Contents Executive Summary... 2 Event Goal(s)/Outcomes... 2 Type of Evidence Gathered... 2 Number of Attendees and Response Rate... 2 Method for Analyzing and Interpreting the Evidence... 2 Results - Participant Learning Outcome Assessment... 3 Results Summary of Daily Evaluations... 3 Results Summary of Overall Institute Evaluation (administered on Day 3)... 5 Additional Conclusions and Discoveries... 6 Use of Results and/or Program Modifications... 7 Reflections on the Assessment Process... 7 Appendix A. Results from the Post Institute Evaluation Surveys... 8 Day 1 Survey Results... 9 Day 2 Survey Results Day 3 and Overall Institute Survey Results... 12

3 Assessment Leadership Summer Institute Executive Summary The Assessment Office (AO) hosted its fifth annual Assessment Leadership Institute (ALI) at Kuykendall Hall from May 17 th 19 th, faculty from 6 schools/colleges and 6 departments participated in the 3-day intensive workshop series designed to create a cadre of faculty to be willing, able, and ready to assume a leadership role in program learning assessment. The Institute featured interactive, hands-on sessions that provided participants with tools and techniques to help their faculty colleagues use assessment to improve teaching and learning. In general, results indicated that the 2017 Assessment Leadership Institute was a success. All participants (100%) exceeded each of the five learning outcomes. 100% of respondents reported the Institute as being very useful, worthwhile to attend, and would recommend the Institute experience to colleagues. At the end of the Institute, each participant successfully presented an assessment project for the academic year and planned three follow-up support group meetings to be held during the fall 2017 semester. Event Goal(s)/Outcomes The learning outcomes of the Institute are as follows: The participants will be able to: (1) Identify applicable learning assessment principles and practices to enhance student learning. (2) Develop learning assessment plans to implement in fall 2017 to support program or institutional level assessment. (3) Utilize learning assessment tools to implement assessment plans in a manner that is meaningful and manageable. (4) Identify venues and locate resources to develop scholarship in teaching and learning utilizing learning assessment. (5) Employ facilitation techniques to guide discussions and collaborative projects. Type of Evidence Gathered The Assessment Office distributed three electronic daily evaluation surveys at the end of each day, which consists of a total of 20 questions to each participant (4 questions on day 1, 4 questions on day 2, and 12 questions on day 3). Number of Attendees and Response Rate All of the ten participants completed day 1 of the post-institute evaluation survey (100%), and nine participants completed day 2 and 3 of the post-institute evaluation surveys (90%). Method for Analyzing and Interpreting the Evidence The Assessment Office used descriptive statistics to summarize closed-ended survey results and carefully reviewed the open-ended responses for future Institute planning.

4 Assessment Leadership Summer Institute Results - Participant Learning Outcome Assessment The Assessment Office set benchmarks for each of the five learning outcomes (see Table 1 for benchmarks and results under each outcome). (Question 2) [N=9] All learning outcomes were exceeded with three outcomes achieving a target of 100%. Table 1. Evaluation Instruments and Results for Each Learning Outcome Outcomes Benchmark Actual Benchmark met? (1) identify applicable learning assessment principles and practices. 80% of the participants report being able to distinguish good assessment practices from bad ones. 100% Exceeded (2) develop learning assessment plans to implement in fall 2017 (3) utilize learning assessment tools to implement assessment plans (4) conduct research on teaching and learning using student learning outcome assessment (5) employ facilitation techniques to guide faculty discussions and collaborative projects 80% of the participants report being able to develop a manageable assessment plan? 80% of the participants report being able to utilize at least 3 good assessment strategies/tools 80% of the participants report being able to locate resources on learning assessment? 30% plan to conduct research or other forms of scholarly activity on learning assessment. 80% report being able to utilize at least 2 facilitation techniques to guide faculty discussions on learning assessment? 89% Exceeded 100% Exceeded 100% Exceeded 44% Exceeded 100% Exceeded Results Summary of Daily Evaluations Usefulness of Topics and Activities (Question 1) Respondents were asked to rank the level of usefulness of the Institute s topics and activities (covered that day). Overall, at least 78% of the respondents found the Institute topics and activities as useful or very useful. In terms of the workshop topics offered at the Institute, at least 89% of the respondents found them useful or very useful. The following topics, offered across the 3 days, were ranked the highest in being very useful by a majority of the participants: SLO Facilitation Activity (90%) Curriculum Map Facilitation Activity (90%)

5 Assessment Leadership Summer Institute Curriculum development through engaging faculty (88%) Meeting Facilitation Lecture and Activity (78%) About 20% of the respondents considered the following activities of little use: Poster presentations (20%) Lunch with poster presenters on Day 1 (20%) Certificate ceremony and Institute reflection (22%) Schedule small group meetings (22%) Most Useful Things Learned to Apply in Assessment Project (Question 2) Respondents were asked, daily, to list two (from Day 1 and 2) or three (from Day 3) things they found most useful that they could apply to their individual assessment projects. Day 1 1. Curriculum map signature assignments (n=5) 2. Facilitation activities (n=2) 3. Reviewing program learning outcomes 4. Relecting on their own program and assessment plans 5. Student learning outcomes activity Day 2 1. Receiving facilitation tips (n=6) 2. Standard setting (n=3) 3. Data analysis and representation (n=2) 4. Rubric adaptation (n=2) Day 3 1. Newly learned facilitation techniques (n=8) 2. Curriculum mapping (n=4) 3. Data analysis and reporting (n=2) 4. Assistance and guidance from other departments 5. Hands-on experiences 6. Scaffolding 7. Terminology clarifications 8. Overall ALI experience What the AO can Continue during the ALI (Question 3) Respondents recommended continue the following practice in the Day1 and Day 2 survey, as well as the Institute evaluation through open-ended comments: Facilitation Activity 1. Assign participants as facilitators and note-takers (n=2) and ensure that everyone participate in the activity (n=2)

6 Assessment Leadership Summer Institute 2. Before the facilitation activity, set the context/stage for the facilitation activity. Explain the activities prior. 3. Facilitating the facilitators (helping out the facilitator along the process) 4. Give feedback on facilitation (facilitator, note taker) Content 1. Maintain the content depth. 2. Use ordinary language to talk about assessment (not terminology). Workshop Format and Management 1. Continue clear objectives. 2. Start the day with introduction and review. 3. Continue well-timed breaks. 4. Combine introduction with activities. (n=2) 5. Invite guest speakers. 6. Allow time for reflection after each activity. 7. Peer review and consultation with Monica and Yao during work time. (n=2) 8. Use of learning strategies with the participants by Monica and Yao. 9. Use interactive activates. 10. Use small group discussion. Institute Organization 1. Continue to provide snacks (n=1) and lunch as a group (n=2). 2. Continue to allow pairs from the same unit. 3. Maintain the cohort size. How the AO can Improve (Question 4) Respondents gave specific suggestions to improve the next Institute on Day 1 and Day 2 s survey, which can lead to the following actions: 1. Explain the activities more in depth prior to the Institute. 2. Remind people more about bringing a sweater to the Institute. 3. Present meeting facilitation module on the first day. 4. Remind people to stop discussion while the speaker is presenting. 5. Give participants time to read SLOs and course descriptions in the Curriculum Map facilitation activity. 6. Give 5 more minutes for the data reporting activity. 7. Give more time for presentation preparation. Results Summary of Overall Institute Evaluation (administered on Day 3) Additional recommended topics and activities included: o How to assess student learning in individual courses o Assessment strategies for millennial students

7 Assessment Leadership Summer Institute o Strategies for exit interview and/or long-term tracking of program graduates as method to assess program value (Question 4) In general, majority of participant s responses considered each aspect of the Institute as very good or good. (Question 5) [N=9] o 56% of the respondents reported that the Institute s application procedure was very good. o 56% of the respondents reported that the communication before the Institute started was very good. o 78% of the respondents reported that the organization of the topics was very good. o 78% of the respondents reported that the usefulness of the materials provided was very good. o 89% of the respondents reported that both Monica and Yao s presentation skills were very good. o 78% of the respondents reported that the interactive activites were very good. 100% of the respondents reported the overall Institute as very useful. (Question 6) [N=9] All of the respondents (100%) reported that they would recommend the Institute to colleagues. (Question 7) [N=9] All of the respondents (100%) believed the Institute was worthwhile. (Question 8) [N=9] 56% reported that the provided lunch helped them commit to the Institute. (Question 9) [N=9] Before the Institute, Yao met with each participant or talked with them on phone to prepare them for the facilitation task(s) that they signed up for. The respondants generally agreed this helped them carry out facilitation tasks for the Institute. Specifically, it mentally prepared the participants for the task ahead and increased their confidence. (Question 10) [N=8] Overall, respondents were very positive about their experience with the Institute: (Questions 8 and 12) o Very worthwhile for reasons previously mentioned. Would recommend to colleagues and other departments. o Very inspiring and it is also good to meet others going through the same journey Please see Appendix A for detailed results. Additional Conclusions and Discoveries The Institute s success is evidenced by several key factors: all learning outcomes were exceeded (with four outcomes achieving a target of 100%), all participants (100%) rated the Institute as useful and worthwhile, and 100% would recommend the Institute experience to colleagues. Furthermore, by the end of the Institute, 100% of the participants developed and presented a realistic, meaningful, and actionable assessment project plan that involved an element of encouraging faculty engagement, collaboration, buy-in, and/or ownership in the assessment process (SLO 2).

8 Assessment Leadership Summer Institute Use of Results and/or Program Modifications The Institute s general content and format will remain the same because of the positive evaluation of the Institute. The Assessment Office will consider making small modifications based on the comments and constructive feedback. Reflections on the Assessment Process The post Institute evaluation surveys and assessment project plan presentations were adequate to collect Assessment Leadership Summer Institute evaluation data. They provide sufficient evidence to prove the success of the institution and actionable results to plan for next year s institution.

9 Assessment Leadership Summer Institute Appendix A. Results from the Post Institute Evaluation Surveys 1. Please rate the usefulness of the Institute topics and activities (topics for Day 1, 2, and 3) # Topic/Activity Not at all A little Useful Useful Very Useful Total Res Useful 7 Day 1: Curriculum map facilitation activity 0% 0% 10% 90% 10 4 Day 1: SLO facilitation activity 0% 0% 10% 90% 10 3 Day 1: Program Assessment Introduction 0% 0% 20% 80% Day 2: Meeting facilitation lecture and activity 0% 0% 22% 78% 9 21 Day 3: Curriculum development through 0% 0% 13% 88% 8 engaging faculty 8 Day 1: Direct assessment, capstone & signature 0% 0% 30% 70% 10 assessment lecture and activity 1 Day 1: Introduction to the Institute 0% 0% 30% 70% Day 2: Resources and scholarship 0% 0% 33% 67% 9 11 Day 2: Rubric adaptation activity 0% 0% 33% 67% 9 10 Day 2: Sharing assessment plans and peer 0% 0% 33% 67% 9 review 23 Day 3: Assessment project presentations 0% 0% 33% 67% 9 5 Day 1: Poster presentations 0% 20% 20% 60% Day 2: Assessment plans working time 0% 0% 38% 63% 9 15 Day 2: Data analysis and reporting lecture and 0% 11% 33% 56% 9 activity 14 Day 2: Lunch with poster presenters 0% 11% 33% 56% 9 13 Day 2: Poster presentations 0% 11% 33% 56% 9 12 Day 2: Standard setting lecture and activity 0% 0% 44% 56% 9 20 Day 3: Conduct actionable assessment use of 0% 0% 44% 56% 9 results presentation 9 Day 1: Institute reflection 0% 0% 38% 63% 8 24 Day 3: Certificate ceremony and Institute 0% 22% 33% 44% 9 reflection 22 Day 3: Schedule small group meetings 0% 22% 33% 44% 9 19 Day 3: Time to work on assessment project 0% 0% 56% 44% 9 6 Day 1: Lunch with poster presenters 0% 20% 40% 40% 10

10 Assessment Leadership Summer Institute 2 Day 1: Participants' self-introductions 0% 0% 60% 40% 10 Day 1 Survey Results Note: Presented in order of highest ranked Very Useful topics. May not sum to 100% due to rounding errors. 2. List two things that you found most useful that you intend to apply in your assessment project curriculum mapping 2. facilitation activities 2. The SLO activity gave me something to think about regarding our PhD program. Learning about direct and indirect evidence. 3. Using the strategies that were discussed in our group to develop curriculum mapping. Using the map to help develop the capstone project Signature assignments. 2. Assessment introduction. 5. Curriculum Mapping Signature Assignment 6. Program Learing Outcome Assessment and facilitation activities. 7. Curriculum map And reviewing PLOs The curriculum map facilitation activity provided a lot of insight on how to engage our faculty in assessment, which is one of the aspects of our project. Both the activity itself and the guidance given to the facilitator were applicable. 2. SLO facilitation highlighted some issues with our PLOs that we can readily address. 9. Curriculum mapping Refining SLO's Collaborative model in developing program, course, and individual assessments. 2. Assessing our program with rubric for assessing quality of program level student learning outcome and assessment plan to develop next steps for our department. 3. What did we do well that should be continued? 1. Everything 2. Having participants be facilitators and note takers for the activities. Having everyone participate during discussions were good too. 3. Translate terminology into ordinary language -- will be useful to get faculty onboard. 4. Allow time for reflection after each activity 5. Getting everyone involved with activities. 6. I like the combination of introduction and some of the activities. 7. Facilitating the facilitators is really helpful to those of us who will be dealing with the rest of the faculty in our units. The cohort size is good and the depth so far is at the right level. Snacks and food help keep us going. 8. Nice pace and good mix of interactive activities Focused topics (introduction and review) followed by well-timed breaks 2. Clear objectives 3. Identification and demonstration of learning strategies used by Yao and Monica throughout the sessions. 4. Pairs (participants) from departments to start collaboration and support carryover 5. Small group discussions.

11 Assessment Leadership Summer Institute 4. How can we improve? 1. N/A 2. Give participants time to read SLOs and course descriptions before they participate in curriculum mapping activity 3. Explain the activities a little more in depth prior to the Institute 4. Classroom very cold 5. Sometimes hard to hear speakers due to discussions in room (naughty students!) - maybe remind folks to keep quiet a bit more. 6. I can't think of any improvements at this time. Excellent!

12 Assessment Leadership Summer Institute Day 2 Survey Results 2. List two things that you found most useful that you intend to apply in your assessment project. 1. Facilitation tips and standard setting Rubric adaptation (need rubric for research process). 2. Meeting facilitation Group feedback on assessment project. We incorporated all feedback in the development of steps. 2. Facilitation tips to engage faculty in collaborative decision-making. 4. Utilizing the facilitation suggestions and attempting to do data analysis and reporting to department. The resources were also helpful. 5. Rubric adaptation Meeting facilitation tips 6. -Meeting facilitation - definitely need organization, planning, and techniques to deal with our faculty - Standard setting 7. We will utilize some of the meeting facilitation tips in our upcoming meetings that we will need to have with our faculty and will also keep in mind some of the data presentation ideas discussed. 8. Standard setting and data analysis activities. 3. What did we do well that should be continued? 1. Peer review and input from institute instructors during working time I was relieved that Monica set the context/stage for the facilitation activity. This served as a preview for the participants. It also freed me as the facilitator to focus on practicing facilitating tips. 2. Yao - thank you for the feedback on facilitation. I'll strive to improve with the tips provided. 3. I appreciated lunch as a group because it allowed participants to brainstorm and network. 3. The table activities and feedback from Monica and Yao during the table activities. 4. Explain the activities prior 5. Facilitation, keeping on time 6. Interactive activities 7. Providing lecture followed by activity. 4. How can we improve? 1. Can't think of anything at the moment! 2. I would have preferred 5 minutes more for reporting data activity. I would have used that time to identify and draft options before selecting a data reporting system. Thank you for another great learning day! 3. I'm finding it a bit hard to convert the project idea to a presentation on the required timescale, we seem to be changing our project a lot. Not sure what to improve, maybe need a bit more pre-planning for the presentation. 4. Meeting and facilitating activity should be on the first day it will help facilitators with activities.

13 Assessment Leadership Summer Institute Day 3 and Overall Institute Survey Results 2. As a result of the 2016 Assessment Leadership Institute, # Question Yes Not Sure No Total Respon ses 1 I am able to distinguish good program learning assessment 100% 0% 0% 9 practices from poor ones? 2 I am able to utilize at least 3 assessment strategies/tools? 100% 0% 0% 9 3 I am able to develop a manageable assessment project 89% 11% 0% 9 plan? 4 I am able to locate resources on learning assessment? 100% 0% 0% 9 5 I plan to conduct research or other forms of scholarly 44% 56% 0% 9 activity on learning assessment? 6 I am able to utilize at least 2 facilitation techniques to guide faculty discussions on learning assessment? 100% 0% 0% 9 3. List three things that you found most useful that you intend to apply in your assessment project. 1. Curriculu map Scaffolding Facilitate meetings 2. Curriculum map Rubric Facilitation techniques 3. Facilitation assessment meetings to engage faculty Curriculum mapping to link courses to SLOs 4. Doing the rubric facilitation allowed me some practice. My assessment goal is develop a rubric for grading our PhD program Qualifying Exam, so I need all the practice I can get speaking to a group of faculty. Also, the data analysis and reporting exercise was a good way to get ideas on different ways to create visuals for data. I will use what I've learned about reporting data when I share information to faculty. I think this will help faculty consider my project more seriously Using pictures as a tool for faculty engagement 2. Feedback and guidance from instructors on project 3. Areas that influence assessment results 6. 1.Getting hands on experience with the Faculty Assignment, it helped me think of creative ways to involve faculty in program/student assessments. 2. Thinking of an assessment project for our department, ALI has helped in the sense of thinking how we are going to incorporate this project and making it successful. Going through ALI has helped me realized no matter how big or small/ challenging an assessment is it is doable and now we have the tools and strategies to implement. 3. Talking with other departments and seeing their previous projects or upcoming projects and how they are implementing has been helpful. We already started using ideas from Dental Hygiene to incorportate into our curriculum assessment. 7. Facilitation exercises and discussion of experiences dealing with challenging faculty Examining issues with curriculum maps and working out how to improve them Seeing other units assessment programs and how successful they can be 8. Utilizing assessment data to make changes, different techniques to engage faculty in our planning process, looking critically at assessment data Distinction between primary and secondary evidence (I used to think student evals counted as primary). 2. Idea that assessment drives programatic change; not just tactical decision-making. 3.

14 Assessment Leadership Summer Institute Techniques for engaging faculty (e.g., I'm using tactic of echoing/acknowledging concerns more often since workshop). 4. Please recommend additional topics/activities for us to provide in the future. 1. How to assess student learning in individual courses 2. Wold "Assessment Strategies for Millennial Students" be any different from what we learn? 3. N/A ALI was a great experience very well thought out and organized. 4. Strategies for exit interviews and/or long-term tracking of program graduates as method to assess program value? 5. Please evaluate different aspects of the Institute. # Option Very Poor Poor Fair Good Very Good Total Res. 1 Application procedure 0% 0% 0% 44% 56% 9 2 Communication before the Institute 0% 0% 11% 33% 56% 9 started 3 Organization of the topics 0% 0% 0% 22% 78% 9 4 Usefulness of the materials 0% 0% 0% 22% 78% 9 5 Monica's presentation skills 0% 0% 0% 11% 89% 9 6 Yao's presentation skills 0% 0% 0% 11% 89% 9 7 Interactive activities 0% 0% 0% 22% 78% 9 8 Lunch 0% 0% 0% 44% 56% 9 9 Snacks 0% 0% 11% 22% 67% 9 6. Rate the overall usefulness of the Assessment Leadership Institute. # Answer Response % 1 Not useful at all 0 0% 2 Of little use 0 0% 3 Useful 0 0% 4 Very useful 9 100% Total 9 100%

15 Assessment Leadership Summer Institute 7. Would you recommend this Institute to colleagues? # Answer Response % 1 Yes 9 100% 2 No 0 0% 3 Not sure 0 0% Total 9 100% 8. Was this Institute worthwhile? # Answer Response % 1 Yes 9 100% 2 No 0 0% 3 Not sure 0 0% Comment to Question 8: Total 9 100% Very worthwhile for reasons previously mentioned. Would recommend to colleagues and other departments. 9. Did the following help you commit to the Institute? Commitment means: attending the Institute, actively engaging in activities; planning to join the support group meetings in the fall; and planning to present at the poster session in spring. # Option It certainly helped me to commit 1 The tablet computer that I will be getting Helped somewhat I would commit regardless Total Responses 0% 22% 78% 9 2 Provided snacks 22% 22% 56% 9 3 Provided lunch 22% 33% 44% Yao explained the facilitation tasks and sent you preparation materials before the Institute (in person or on the phone). In what ways did this help or did not help you carry out the facilitation tasks assigned to you? 1. It helped to get an idea of what the activity would be about but I feel that having an additional meeting with her after I had time to read over the materials would be helpful 2. It helped, in particular by making me start reading the materials ahead of the institute. 3. I met with Yao beforehand, and it helped lessen my anxiety. During the institute, the rubric adaptation facilitation script was a little hard to follow...there was a lot to read and I got a little confused, but it turned out fine. 4. Explanation prior to the ALI was helpful in setting exceptions. It also ramped up (some) anxiety. Consider sharing facilitation and note taking techniques in group forum at the beginning of ALI, since some individuals were not facilitators.

16 Assessment Leadership Summer Institute 5. Talking over the phone with Yao was extremely helpful. It helped me understand my assigned tasks and duties for the activities. 6. I felt much more prepared since the materials without the discussion didn't make that much sense to me. 7. It made me think about it ahead of time much more than I would have without it. 8. It was helpful to have the scenario ahead of time. I was a bit concerned by the time spent in preparation on dealing with recalcitrant faculty; in the actual activity, everybody was (perhaps unrealistically) reasonable! 11. The length of the 3-day institute is: # Option # Responses Response % 1 Too short 0 0% 2 Just right 8 89% 3 Too long 1 11% 12. Any other comments related to your Institute experience? 1. I found that short presentations from previous presenters as well as the guest speaker on the last day was extremely useful 2. The institute was a great experience and I learned a lot. I still feel like there is a lot of information that I have not fully absorbed yet, but I can always look in my binder or at the documents on Laulima. Yao and Monica, and their assistants were extremely organized and incredibly generous with their expertise. I feel very privileged to have been a part of the institute. I hope I can make a positive assessment contribution to our program. Thank you! 3. I thought the breadth and depth of material was appropriate for the 3-day institute. Introduction of new information was well paced with breaks for discussion and assimilation. Thank you! 4. Wonderful wondering is there is an ALI 1.2/102/next level of assessment learning program to take. 5. Very inspiring and it is also good to meet others going through the same journey 6. This is a bit delicate -- asking Dean for $200 per participant is perfectly reasonable and a great value. However, it was a bit unexpected. I think the cost is very appropriate, but I didn't see it mentioned when I applied, which was a bit confusing.