Increase Survey Response Rates

Increase Survey Response Rates Drive visibility and credibility. A successful learning analytics program relies on the ability to collect data. The higher the response rates for your training surveys, the more confidence you can have in the accuracy of the data. We hope you find this to be a valuable guide to training industry best practices for increasing survey response rates. White Paper Metrics That Matter

Contents Measuring Response Rates & Setting Goals... 3 Recommended Response Rates for Analysis... 4 Develop Survey Content with Consideration... 6 Provide Ease of Access... 6 Accessing the Survey... 6 Surveys for Instructor Led Training... 7 Surveys for Self-Paced Web-Based Training... 8 Utilize Communication... 9 Use Reinforcement... 10 Positive Reinforcement... 10 Negative Reinforcement... 10 Closing Thoughts... 11 About CEB... 11 About Metrics That Matter... 11 Contact Us to Learn More... 11 2015 CEB. All rights reserved. 2 www.cebglobal.com/mtm

Measuring Response Rates & Setting Goals Metrics That Matter Tool In Metrics that Matter you can run reports to track response rates by navigating to Reporting > Show All Reports > Response Rates. Note that for this report to be meaningful, you must store class enrollment data in MTM. In order to improve response rates, the first step is to understand the percentage of students who are submitting surveys. When measuring response rates, begin by separating the data into categories by survey type and delivery method. Determine what your response rates have been historically for each of these categories and how they are trending. Establish monthly checkpoints to measure and analyze response rates. After instituting any new programs to boost response rates, perform this measurement analysis weekly. After gathering the data, it is important to set goals for response rates. You will certainly set different goals by survey type and delivery method. You might also set separate goals for your recurring learning events versus your strategic, visible, or costly programs. In setting your goals, ask the following questions: What response rate do we feel is needed for reasonable conclusions to be drawn for day-to-day learning measurement? What response rate do we feel is needed for a more in-depth measurement exercise? How much money and how many resources are we willing to commit to increasing response rates? Does management require minimum response rates? Survey Type Paper Online Email Post Event 82% 59% 53% Follow Up 33% 38% 44% Manager 25% 26% 31% Figure 1: Average Response Rate by survey type and collection method Survey Type Recurring Learning Events Strategic/Visible/ Costly Programs Post Event 70% 78% Follow Up 54% 67% Manager 53% 66% Figure 2: Response rates desired by organizations for meaningful analysis and decision making 2015 CEB. All rights reserved. 3 www.cebglobal.com/mtm

Recommended Response Rates for Analysis Metrics That Matter recommends establishing thresholds for different measurement needs. The following provide some basic guidelines to follow based on different reporting needs and scenarios where data can provide valuable decision support. Key Takeaway Statistical significance should never outweigh business significance. Statistical significance should be used as a guideline for the point at which additional surveys are unlikely to shift the ratings higher or lower. To Manage L&D Operations or a Program: All programs should collect surveys from 65% of the learner population. If rates are below 65% or established goals, target an increased participation rate of 10% annually. To Manage Courseware: When conducting analysis of overall performance, aim for at least 500 surveys. For analysis of any specific course, collect at least 30 surveys and achieve a 65% response rate or higher. If rates for individual courses are below 65% or established goals, target an increased participation rate of 5% annually per course. To Guide Enrollment Decisions or Correct Issues: Even a single survey can be used to provide an anecdote of success or an opportunity for improvement. Consider the value of star ratings and comments of users in an app store or when purchasing books online, this kind of information can help learners to choose learning activities and set expectations. On the other hand, a single piece of constructive feedback provides an opportunity to reach out to a learner to understand an issue further or fix a negative experience. To Market Training Value: In order to demonstrate value for a particular course or to a specific audience (i.e. internal/external client groups), collect at least 30 surveys and achieve a 65% response rate or higher. To Report with Statistical Significance: The table below reflects the number of surveys needed to achieve a 2.5% or 5% margin of error based on a specific number of learners. 2015 CEB. All rights reserved. 4 www.cebglobal.com/mtm

2.5% Margin of Error 5% Margin of Error # of Learners # of Surveys Response Rate # of Surveys Response Rate 10 10 100.0% 10 100.0% 20 20 100.0% 19 95.0% 30 29 96.7% 28 93.3% 50 48 96.0% 44 88.0% 75 72 96.0% 63 84.0% 100 94 94.0% 80 80.0% 150 137 91.3% 108 72.0% 200 177 88.5% 132 66.0% 250 215 86.0% 152 60.8% 300 251 83.7% 169 56.3% 400 318 79.5% 196 49.0% 500 377 75.4% 217 43.4% 600 432 72.0% 234 39.0% 700 481 68.7% 248 35.4% 800 526 65.8% 260 32.5% 1000 606 60.6% 278 27.8% 1200 674 56.2% 291 24.3% 1500 759 50.6% 306 20.4% 2000 869 43.5% 322 16.1% 2500 952 38.1% 333 13.3% 3500 1068 30.5% 346 9.9% 5000 1176 23.5% 357 7.1% 7500 1275 17.0% 365 4.9% 10000 1332 13.3% 370 3.7% 25000 1448 5.8% 378 1.5% 50000 1491 3.0% 381 0.8% 75000 1506 2.0% 382 0.5% 100000 1513 1.5% 383 0.4% 250000 1527 0.6% 384 0.2% 500000 1532 0.3% 384 0.1% Figure 3: Statistical significance sample size and response rates required for a confidence interval of 95% 2015 CEB. All rights reserved. 5 www.cebglobal.com/mtm

Develop Survey Content with Consideration When looking to increase response rates, the primary goal for the content of your survey should be to make it as easy as possible for respondents to complete it. To achieve this goal, consider the following approaches: Key Takeaway Surveys should be designed to measure only the critical aspects you need to manage. For shorter classes and training that is not designed to change behaviors or impact business results, consider an abbreviated survey. Streamline your surveys by removing questions that will not be used in reporting. For help with this, consult with your Metrics That Matter account team. For shorter training events, use an abridged version of your primary survey that asks only the essential questions. Review each of the questions on the survey to ensure that they are clear, concise, and easy for a student to respond to. Provide the surveys in the primary languages of all your learners. If multiple languages are used within a particular audience, consider using the multi-language versioning functionality which is available in Metrics That Matter. Accept anonymous survey submissions for surveys. Note that this could provide a negative impact for Follow Up surveys, if email addresses are not collected for the anonymous submissions. Provide multiple formats and entry points for collecting surveys. (See Appendix D for the Metrics That Matter Standard and Smartphone experiences.) Provide Ease of Access The most effective way to boost response rates of surveys is to provide easy access and appropriate time for the respondents to complete surveys. Accessing the Survey When completing the surveys using Metrics That Matter, respondents can reach the survey through several access points: Company URL (with class dropdown) Quick Response (QR) Code Custom URL Email directly to respondent MTM Survey MTM Mobile 2015 CEB. All rights reserved. 6 www.cebglobal.com/mtm

Access Type Benefits Requirements/Consid erations Email directly to Respondent Company URL Custom URL Best for tracking and automated reminders. Emails are student specific. Simple. Can be provided verbally. Simple. Can be provided verbally. Can be specified by Instructor, Location, Learning Method. Learners enrolled via upload or LMS import. Must have email access in class. Respondent selects class from list. Not practical for companies with high training volume. Learners must choose from list; not practical for Locations or Learning Methods with high training volume. Class Summary URL or QR Code MTM Mobile Application Links directly to survey for class; no need for respondent to select class. Optimized for use with smartphones. Provides easy in-class survey completion experience Difficult to provide verbally. QR Codes require smartphones with a scanner application. Learners must be enrolled via upload or LMS import. Requires learners to download MTM app. Surveys for Instructor Led Training Instructors should notify students at the beginning of class that they will be asked to complete surveys and that they provide an opportunity to express their feedback on the event. Instructors should also note that the feedback is used to improve the quality of the training for learners and share an anecdote of an action that was taken as a result of prior feedback. On the last day of class, the instructor should re-communicate the importance of completing the surveys before providing students with the URL, QR Code, emailed link and/or paper surveys (see Appendix A: Sample Instructor Script). Additionally, instructors can hand out a brief information sheet regarding the survey (Appendix B: Sample Training Survey Notice). Time to complete the surveys should be provided midway through the last day or at the last break of class, rather than at the very end of class when learners are anxious to leave. This allows the learners to provide thoughtful feedback without feeling rushed. The instructor should leave the room for about 10 2015 CEB. All rights reserved. 7 www.cebglobal.com/mtm

minutes to allow the students to submit candid surveys. If collecting surveys on paper, a student should be designated to collect the surveys. In classrooms with computers and Internet connectivity or with learners who have access to smartphones and mobile devices, Metrics That Matter offers a number of ways to access the survey (see below). For classrooms where no Internet access is available, the two options are to 1. Send invitations to respondents via email to complete the survey, or 2. Collect paper surveys If your organization is utilizing email invitations, you can customize the subject and body of the email message to make it unique to your organization. Communicate the importance of the respondents feedback in the message, set expectations for time needed to complete, and provide the details of any positive or negative reinforcement (see Appendix C: Sample Emails). Metrics That Matter can also automate one or more reminder emails to be delivered to respondents who do not complete assigned surveys within a set number of days. While respondents may ignore the first email, they are less likely to take no action after receiving two or three. Further, the emails can include a carbon copy to the student s manager, if enrollment and manager information is uploaded or passed via your LMS integration. Surveys for Self-Paced Web-Based Training For Self-Paced Web-Based (elearning, WBT) training, common delivery methods for the survey include: 1. Send survey invitation emails to the learner. 2. Embed a link to the survey in the elearning content or learner portal. 3. Embed a redirect link to navigate the learner to the survey upon completion. 4. Combination of the above, such that the student may complete the survey within the learning experience, but will be prompted to do so in an email if they do not. Since many students do not typically complete elearning tutorials, it s best to send surveys to all students who enroll in the training, as opposed to surveying only completions. The ideal logic to establish for this is to send out emails to all students who complete the training and all students who enrolled for the training a set number of days ago, but have not yet completed. The number of days will be dependent on the typical time to complete the tutorial. 2015 CEB. All rights reserved. 8 www.cebglobal.com/mtm

Utilize Communication To boost survey response rates across an organization, the main idea that needs to be impressed upon the learners is that it is important for them to complete the surveys. There are a few different approaches to this that should be used in conjunction with one another: Metrics That Matter Tool In Metrics that Matter, you can add certificates with your organization or Learning department logo and a signature. Custom certificates are also available contact your support representative for more information. When students are in class, the instructor should communicate at the start, middle, and end of class that they will be receiving a survey. The instructor should explain how it will be delivered and why it is vital that the students complete it. See Appendix A: Sample Instructor Script for a sample instructor script. Provide another in-class communication in the form of a handout that details the information regarding the training surveys. See Appendix B: Sample Training Survey Notice for an example. Following training, the instructors can personally follow up with all class attendees or specifically target those who have not submitted surveys. If surveys are being accessed through automated email messages, customize the email messages to make them unique to your organization and communicate your message for why the learners should complete the survey. Use and customize automated reminder emails to go to learners who ignore the first survey invitation email. See Appendix C: Sample Emails for examples of the default emails. Share the results of the data collected back to learners, so that they understand that the few minutes they take to complete the survey are time well spent. When sharing the survey data with learners, be sure to tie it back to job impact and business results. Ideally, you can show the improvement over time in these areas and give the learners credit for helping you make those improvements. Gain buy-in and support from leadership. If learners are being told to complete the surveys not just by the training organization, but also by their managers, response rates are likely to increase. Spur competition among instructors or business units by publishing comparison response rates and rewarding top performers. Instructors are more likely to communicate the importance of completing the survey, if they know that they are accountable for the class response rate. Leaders of business units can be motivated by viewing their group s ranking among the others. 2015 CEB. All rights reserved. 9 www.cebglobal.com/mtm

Use Reinforcement Often, response rates will be impacted significantly when organizations use reinforcement techniques effectively. Positive Reinforcement Many organizations present incentives for respondents who submit surveys. These positive reinforcements will only boost response rates if the respondents are aware of them, so ensure that your communications regarding surveys advertise them. Provide certificates of completion to all students who submit surveys Show Learners Course Ratings - Metrics That Matter offers three options for displaying star ratings (5 star scale) to learners, increasing visibility of applications of the survey process: o o o Display the results in the user experience immediately after a Post Event survey is submitted Pass star ratings and testimonials into your LMS or learner enrollment portal via integration with MTM Use the MTM Mobile app to allow users to search and review ratings and testimonials for courses Provide a prize to a randomly selected student from each class or over a period of time. Prizes that might be appropriate include: o o o o Gift certificates (coffee houses, stores, online shopping) Small Electronics/Gadgets Bonus vacation time Discounts on future training (for commercial training) Negative Reinforcement An effective means of increasing response rates can be instituting negative consequences for those who do not complete surveys. Apply negative reinforcement carefully. If students resent being forced to submit surveys, the quality of the comments and potentially even the scale-based answers could suffer. Metrics That Matter Tool When your LMS is integrated with Metrics that Matter, completions can be passed back into the system for automated completion marking. Withhold credit/completion marking for a course until the survey is submitted Inform managers of learners who have not submitted surveys by including them on carbon copies of emailed reminders 2015 CEB. All rights reserved. 10 www.cebglobal.com/mtm

Closing Thoughts Metrics That Matter SmartSheets enable organizations to quickly implement a robust, credible, automated, action-oriented approach to learning analytics. They enable organizations to stand upon over a decade of research and refinement with the confidence that the measures are valid and proven. While other organization use their smile sheets and have endless debates over the wording of questions, organizations that adopt the automated SmartSheets of Metrics That Matter are able to redirect their energies toward taking action: improving the business impact of their learning programs and communicating forecasts and results to their business stakeholders. About CEB CEB is the leading member-based advisory company. By combining the best practices of thousands of member companies with our advanced research methodologies and human capital analytics, we equip senior leaders and their teams with insight and actionable solutions to transform operations. This distinctive approach, pioneered by CEB, enables executives to harness peer perspectives and tap into breakthrough innovation without costly consulting or reinvention. The CEB member network includes more than 16,000 executives and the majority of top companies globally. About Metrics That Matter The standard in learning analytics, Metrics That Matter from CEB enables more precision in strategic talent decisions, moving beyond big data to optimizing your workforce learning investments against the most businesscritical skills and competencies. Our cloud-based talent analytics software and proven methodology enable organizations to run learning and talent like a business by establishing the right metrics, comparing against external benchmarks, and leveraging automated insights. Over 500 organizations utilize Metrics that Matter, including the most widely recognized organizations for talent development. Contact Us to Learn More Phone: +1-800-561-3341 or +1-312-676-4400 Email: MTMsolutions@executiveboard.com Web: www.cebglobal.com/mtm 2015 CEB. All rights reserved. 11 www.cebglobal.com/mtm

Appendix A: Sample Instructor Script Instructors should take time at the beginning, middle, and near the end of class to explain the survey process and how important it is for the learners to complete surveys. Below, there are sample scripts that instructors can read from in class. These should be customized to reflect your organization s survey process and any positive or negative reinforcement associated with completing the surveys. At the beginning of class: At the end of class, you will be asked to submit a brief survey regarding this training. The survey will not only measure how satisfied you were with the instruction and courseware, but also how relevant the class is to your job. This feedback is vital to us as we want to continue to improve the quality and usefulness of the training that you attend. On the last day of class, when students are completing surveys in class: Please take the next 5-10 minutes to complete the survey. Your feedback is extremely important, because we use this information not only to improve our training but to have a meaningful discussion with your management on the value of the training to you and the business. On the last day of class, when students will complete surveys via email links: Very soon, you will receive an email inviting you to take a survey regarding this class. It will take about 5 minutes to complete. Please complete it shortly after receiving the email because your thoughts on the training will be fresh and so you won t receive reminder emails later. Your feedback is extremely important, because we use this information not only to improve our training but to have a meaningful discussion with your management on the value of the training to you and the business. On the last day of class, when students will complete Follow Up surveys: About two months from today, you may receive a Follow Up survey. This brief survey will focus on if and when you applied the training to your job. This survey is critical to understanding the effect of the training in practice. We would appreciate it if you spent a few minutes of your time to complete that survey, so that we can make sure that future training sessions have a positive impact on your job success. 2015 CEB. All rights reserved. 12 www.cebglobal.com/mtm

Appendix B: Sample Training Survey Notice You can customize this notice to be consistent with your process for collecting surveys. If you have any positive or negative reinforcement techniques associated with your survey process, be sure to include those in this notice. Ensure the final notice is visible in each student s courseware materials. All instructors should be aware of the notice in the materials and reference the importance of completing the survey as well as the process by which students will receive the survey. Training Survey Notice Thank you for attending training. Your candid and objective feedback is important to us. We use this information to improve the training and ensure it is useful to you on the job. Why? Your feedback is extremely important, because we use this information not only to improve our training but to have a meaningful discussion with your management on the value of the training to you and the business What? The survey is taken very seriously by our organization to understand your satisfaction, your opinion on learning effectiveness and most importantly, it provides your management and ours information on the impact it will have on the job and on business results. When? An electronic survey will be emailed to you following this training. It will take about 5 minutes to complete and it will help us most if you complete it within 48 hours of receiving it. How? You complete the survey online as it will be sent via email. Simply click on the link and begin the survey on any web-enabled device. Follow Up Survey About two months from today, you may receive a Follow Up survey. This brief survey will focus on if and when you applied the training to your job. This survey is critical to understanding the effect of the training in practice. We would appreciate it if you spent a few minutes of your time to complete that survey, so that we can make sure that future training sessions have a positive impact on your job success. Thank you again for your time in attending the training and your consideration in completing this important survey. [Signature and Brand elements] 2015 CEB. All rights reserved. 13 www.cebglobal.com/mtm

Appendix C: Sample Emails You can customize the subject and body of email messages to make them fit within your organization. Be sure to communicate the importance of the students feedback in the message and provide details of any positive or negative reinforcement. Post Event Email From: Company Name Subject: Please provide feedback about training you attended Recently, you attended the class, [Coursename]. We would appreciate it if you could sped a few minutes to provide feedback related to this course so that we may continue to improve the quality of the programs we offer. Please follow the link below to complete the survey. Click here to begin the survey [Signature and Brand elements] Reminder Email From: Company Name Subject: Reminder - Action Requested Several days ago you received an email to complete a brief survey regarding the class, [Coursename]. To date, we have not received your response. In order to continuously enhance the value and effectiveness of the courses offered your thoughtful and timely feedback is essential. Please click on the link below and complete this brief survey. Your input is appreciated. Click here to begin the survey [Signature and Brand elements] 2015 CEB. All rights reserved. 14 www.cebglobal.com/mtm

Follow Up Email From: Company Name Subject: Please provide additional feedback about training you attended A couple of months ago, you completed the training course, [Coursename]. Please click the link below to fill out a brief follow up survey on that class to help us continuously improve. Thank you in advance. Click here to begin the survey [Signature and Brand elements] From: Company Name Subject: Manager 60 Day Follow Up Dear [Manager Name]: Manager Email Several months ago your employee, [Student Email], completed a training program entitled [Coursename]. In order to measure the effectiveness of this training program in developing your employee, please click on the link below and complete a brief survey. Click here to begin the survey [Signature and Brand elements] 2015 CEB. All rights reserved. 15 www.cebglobal.com/mtm

Appendix D: Survey Experience in Metrics that Matter Metrics That Matter supports multiple user experiences for survey collection in order to provide ease of submission, gather timely feedback, ensure high response rates, and support the on-the-go employee. The entire data collection experience is streamlined and optimized for online, smartphone and mobile browsers. Smartphone optimization removes images and styling to allow for faster loading and easy survey completion. If using a smartphone for access, browser detect directs the respondent to optimized experience. When using a device equipped with speech-to-text capabilities, comment questions may be completed without typing. Metrics That Matter Data Collection Experience Email Entry Page Demographics Evaluation Certificate Thank you! ***** Standard Online Survey View 2015 CEB. All rights reserved. 16 www.cebglobal.com/mtm

Smartphone and Mobile Survey View Supported Devices include: ios (6.0+): iphone 4, 4S, 5, 5C, 5S Android (2.2+): Samsung Galaxy S or Note series, HTC One and Evo, and Motorola Droid, Razr, or Moto series Blackberry OS (5.0+): Curve, Bold, Q10, and Z10 Windows Phone (7+) 2015 CEB. All rights reserved. 17 www.cebglobal.com/mtm