Course Evaluations. Jim Farrell Product Manager Users Conference New Orleans March 20-23

Transcription

1 Course Evaluations Jim Farrell Product Manager 2012 Users Conference New Orleans March 20-23

2 Jim introduction My background in educational and instructional technology Worked as a elementary school teacher Worked as an instructional designer Joined Questionmark in 2007 in product management Connect via Linkedin and Twitter (jimpfarrell) Slide 2

3 Your introductions By show of hands: How many people work in corporations? How many work in academic institutions? How many work in government/military? Other? By show of hands: How many people feel currently conduct course evaluation surveys using Questionmark? Slide 3

4 Best practices around course evaluation surveys using Questionmark We will now go into best practices around course evaluation surveys Much of this material is gleaned from the course evaluation best practice guide available to customers: rk.com/perception/help/ v5/best_practice_guides/ course_evaluation/defaul t.htm Slide 4

5 Did they Like It? Course Evaluation Use Questionmark Live and add questions: From library Create your own Add demographic questions to filter and compare: Location Instructor Course Participants answer Course Evaluation Report with Analytics Course Summary Instructor Summary Class detail Class summary Slide 5

6 Slide 6

7 Authoring Course Evaluation Surveys Slide 7

8 Step 1: Course Evaluation Authoring Create a course evaluation survey assessment Choose from a library of questions or make your own Slide 8

9 Slide 9

10 Slide 10

11 Slide 11

12 Slide 12

13 Slide 13

14 Authoring course evaluations Step 2: Use AM to finish the job (import Qpack from Questionmark Live) Set up topics with reporting in mind (e.g., put demographics questions into a demographics topic) Follow the checklist: No random administration of questions Slide 14

15 Step 3: Authoring course evaluations Get the questions and survey reviewed internally Make edits if required Get ready for the next phase: Scheduling and Delivery Slide 15

16 Scheduling and delivery of course Two main options for delivering course evaluation surveys: Open (i.e., open.php) Allows open anonymous access by participants Scheduled Allows you to track which participants took the survey More details in the course evaluation best practice guide Do a pilot run first to make sure everything works the way you expect evaluation surveys Slide 16

17 Using QR Codes to Launch Questionmark Assessments Slide 17

18 The Bar Code as used on a Questionmark Assessment Scanner and/or software reads code Can display and/or save as plain text Slide 18

19 What are QR Codes? QR = Quick Response Type of barcode Allows more data than traditional barcode Used in marketing, advertising, entertainment Increasing use in e-learning/m-learning Slide 19

20 QR Codes are read by Apps iphone Redlaser Neoreader ScanLife Android Google Goggles QR Droid Scanlife Windows Mobile QR Reader BeeTag Quickmark Blackberry ScanLife BeeTag QR Code Scanner Slide 20

21 Embed Demographic info in QR Codes Group Special fields such as Instructor Location Course name What ever you want (limited to 50 characters) Language (for multilingual assessments) Slide 21

22 QR Codes for Launching Assessments Examples Print in a page of a book Embed in a PowerPoint Link to a Course Evaluation Benefits Increasing response rates because Quick and easy to access the right URL (no typing required) Increasing data captured allows Data easily captured (no typing required) Easier analysis of survey results by demographic Slide 22

23 Enables quick launching of quizzes and course evaluations Books PowerPoint Signage Slide 23

24 Looking at Results Slide 24

25 Reporting on course evaluation survey The Survey Report in EM can be run But there are 4 reports in Analytics that are focused specifically on course evaluations: Course summary report Instructor summary report Class summary report Class detail report Also there is a Survey Frequency Analysis report that would be useful too results Slide 25

26 What is Questionmark Analytics? ETL Slide 26

27 Course evaluation reports The four course evaluation reports allow you to drill into more and more detail in terms of course evaluation results Start at the course level and work your way down to an instructor/question level of detail Slide 27

28 Four Course Evaluation Reports: Course Summary Instructor Summary Class Summary Class Detail Slide 28

29 Course Summary Compares evaluation results across courses Useful for managers and supervisors to determine how different courses within an organization compare Dynamically generated table representing the summary survey scores for all topics in each course. Averages are ranked and color coded from lowest to highest score. Slide 29

30 Instructor Summary Compares all instructors within a course in terms of evaluation feedback Dynamically generated table representing summary survey scores for all topics for each instructor Averages are ranked and color coded from lowest to highest score. Slide 30

31 Class Summary Compares evaluation feedback for a single instructor for a single course. Useful for Managers and supervisors to determine how different instructors within an organization compare Instructors to obtain summary information on their course evaluation results. Slide 31

32 Class Detail Provides detailed question level course evaluation information for a single instructor for a single course. Useful for Managers and supervisors to determine how different instructors within an organization compare Instructors to obtain summary information on their course evaluation results. Slide 32

33 Demo of Questionmark Analytics course How about a quick demo of Questionmark Analytics to show you the course evaluation reports? I have set up a typical course evaluation survey and some results to report on evaluation reports Slide 33

34 Thank you for your time, attention, and contributions! Any questions or comments? Questions? Slide 34

35 Appendix A Related additional resources 2012 Users Conference New Orleans March Slide 35

36 Resources Questionmark: Questionmark blog: Course evaluation best practice guide: Analytics knowledge base: Analytics help guide: Books on survey research and design: Alreck, P. L. & R. B. Settle (1985). The Survey Research Handbook. Homewood, Illinois: Irwin. Devellis, R. F. (2003). Scale Development: Theory and Application. Newbury Park, CA: Sage. Fowler, F. Jr. (1993). Survey Research Methods. Newbury Park, CA: Sage. Slide 36

37 Tips on obtaining the best response rates possible for course evaluations Ensure that participants know that their responses are anonymous. This will encourage higher response rates and truthful responses. Encourage participants to provide both good and bad feedback so that the instruction, course materials, facilities, and course curriculum can be improved. Have participants take the survey on the last day of the course, or very soon after the course, so that their perspectives are fresh and genuine. For example, link from the end-of-course exam to the course evaluation survey. Consider completion of the course evaluation survey to be part of the successful completion of the course. Encourage multiple delivery platforms for the course evaluation survey (e.g., mobile as well as traditional desktop/laptop) to make the taking of the survey more convenient for participants. Consider reporting back to participants about summary results from the course evaluations and plans for improving courses in the future. Keep the survey short and sweet so that participants are not fatigued by the process of taking it. Ensure that the survey is free of spelling or grammatical errors. Consider offering an opportunity for course evaluation responders to win a prize (e.g., gift certificate, t-shirt, coffee mug, etc.) for responding within a specified time window. In practice, treating the gathering of survey results as an exchange tends to increase response rates ( In return for taking this survey we will offer you an opportunity to win. ) Slide 37

38 Slide 38