Aspiring For More Than Crumbs: The impact of incentives on Girl Scout Internet research response rates Debra Dodson, Girl Scout Research Institute, GSUSA Meredith Reid Sarkees, Girl Scout Research Institute, GSUSA Catherine VonFange, Abt SRBI The impact of incentives on Girl Scout Internet research response rates
GSUSA Response Rate Experiment Overview Main objective of this 2012 study was to measure customer satisfaction with Girl Scouting and progress on leadership outcomes. Another goal was to have as many girls as possible respond in each phase, a variety of tactics and incentives were utilized to increase both registration and survey response rates. This report summarizes the efficacy of these efforts. The analysis takes into account the unique challenges of a two-step registration-tosurvey process and complications of obtaining prior parental permission. The results are being used to design and develop approaches and incentive plans for Internet and multi-mode surveys for GSUSA.
Caveats This analysis does not reflect an experimental design, but rather retrospective observations of response rates associated with reminders and incentives introduced in the conduct of an existing study. Though the ten councils were not selected randomly, they do reflect the geographic and size range of US Girl Scout Councils.
Councils participating in the Retention Study Black Diamond Greater Los Angeles Citrus North Carolina Coastal Pines Colorado Northeast Texas Connecticut Southwest Texas Greater Chicago and Northwest Indiana Spirit of Nebraska
Two-Step Research Process Step One: Recruitment and Parental Permission Step Two: Survey Completion Under age 13? Over age 13? NEW Participants Sent online Girl Scout Voices Registration Invitation including active consent permission to survey and use data in follow-up research REGISTERED Panelists Sent online Girl Scout Voices Survey Invitation. NEW Participants Sent online Girl Scout Voices Registration Survey including passive consent permission to survey and use data in follow-up research REGISTERED Panelists Sent online Girl Scout Voices Retention Survey Invitation. Online Registration Survey is completed by parent. Girl and parent complete retention survey together or separately Girl now has permission to participate in future surveys administered through Girl Scout Voices. If receive parental request to opt-out daughter from participating, NOT invited to future surveys. If no opt-out notice received, Girl may participate in retention survey. Girl now has permission to participate in future surveys administered through Girl Scout Voices. 5
Starting Point 8,650 girls were already registered to participate in surveys with the councils prior to the registration phase. Connecticut, Colorado, Greater Chicago and Greater Los Angeles were underrepresented prior to the registration phase. Girls third grade and under, Urban girls and Hispanic girls were initially underrepresented.
Hypotheses Tested Reminders: 1. First reminders sent soon after the initial invitation achieve the greatest increase in response rate; subsequent reminders increase total response rates at diminishing rates. 2. Reminders improve the likelihood survey respondents will resemble the targeted population. Incentives: 3. Individual incentives will be more effective than lotteries and contribute to improved data quality. 4. Low cost individual incentives will be less effective in increasing response rates and improving data quality than higher cost incentives.
Registration Response Rate Total Registration Invitations Sent: 201,458 Completed: 1,731.86% Not completed: 199,727 Reminder 1 Reminder 2 Completed: 1,278.64% Completed: 1,807.90% Not completed: 198,449 Not completed: 196,642
Survey Response Rate Total Survey Invitations Sent: 13,466 Completed: 2,105 16% Not completed: 11,361 Reminder 1 Reminder 2 RoboCalls and Parent Letters Reminder 3 Completed: 953 7% Completed: 512 4% Completed: 142 1% Completed: 442 3% Not completed: 10,408 Not completed: 9,896 Not completed: 9,754 Not completed: 9,312
H01: Reminders sent soon after initial invitation achieve greatest increase in response rate: subsequent reminders increase responses at a lower level 201,458 invitations were sent to parents or guardians requesting their assistance in registering their daughters to participate. Once registered, 13,466 invitations to participate in the survey were subsequently sent. Results of H01 testing are mixed and vary within each Step: H01 not supported in Step One Survey Registration. Initial Response Rate :.86% First Reminder:.64% Second Reminder:.90% H01 supported in Step Two Survey Completion. Initial Response Rate : 15.63% First Reminder: 7.08% Second Reminder: 3.80% Robo Call: 1.05% Third Reminder: 3.28%
H02: Reminders improve the likelihood that survey response will more closely resemble targeted populations. Results of H02 testing are mixed and vary within each Step: H02 Step One Survey Registration. Significant improvement in council representation were more likely after the second reminder than after the first. First reminder (but not second) yielded significant increase in urban girl registration. Significant improvements in grade level representation were more common after the first than the second reminder. (Enhanced incentive announcements were also added to the second reminder). H02 Step Two Survey Completion. No significant benefits in data quality.
H03 and 04: Individual guaranteed incentives will be more effective than lotteries and low cost incentives less effective in motivating response rates. Results of H03 and H04 testing are inseparable in this experiment as they were offered together. At both Step One and Two, a lottery promised 40 winners of a $25 prize. At registration Step One participants were offered an individual virtual incentive. At survey Step Two Participants were offered a GSUSA patch. Results of the enhanced incentives appear to be mixed. H03 and H04 provided the following hierarchy of incentives for further testing: Individually earned and awarded incentives that are coveted and have more perceived value, Lotteries for prizes with monetary value, and Virtual incentives with no perceived value. Data quality did not appear to be impacted significantly by incentives.
Conclusions and Recommendations Registration is the more critical step in achieving better GS Voices representation of council populations. Increased time and incentive resources should be applied to this step. Virtual incentives such as those offered will be abandoned because they did not increase participation. Other more advanced virtual incentives may be tested in the future apps, games, etc. The number of contacts and reminders across both steps with emphasis on registration should be increased. Explore incentives that may appeal to ongoing Girl Scout members and those that may not return to Girl Scouts. Evaluate the cost effectiveness of robo calls as a reminder mechanism. Explore strategies that go beyond repeated contacts or incentives to reach persistently under-represented populations of girls, i.e. urban areas or Hispanic girls.
Next Steps Conduct a comprehensive 2013 Retention Survey based on a rigorous Experimental Design across 17 councils. Continue testing myths long held in the organization such as under-representation of rural populations. Focus resources on building stronger population representations at the registration phase.