Survey Router Management:

Transcription

1 Router Management: An Experimental Examination of Impact on Results Nancy Brigham, Ph.D., VP Respondent Access & Engagement Jason Fuller, Director of Analytics Paper presented at the 2012 CASRO Online Conference, Las Vegas, NV

2 Router Management: An Experimental Examination of Impact on Results Router Management: Quality Matters It is now readily apparent that the research industry is facing one of our biggest challenges: sample capacity. The number of respondents who are willing and available to participate in our studies is declining. We see this decline in response rates for all modes of data collection, but it is especially problematic in online research where the demand is ever-increasing. Setting aside respondent engagement factors (such as long surveys, little incentive, poor survey designs), there are several key market factors that have contributed to this challenge, among them the declining numbers of respondents who want to join panels and the rapid growth of online studies (e.g., movement of offline to online, explosion of new DIY surveys). And, exacerbating the issue is the inefficient way we manage the respondent supply we do have. Traditionally, if a respondent does not qualify or is over quotas for a study, or the study is closed when they access it, we simply turn them away from the study, and don t give them a chance to take another survey. This causes frustration for respondents, lost opportunities for researchers and clients, and is a drain on panel/sample management. routers are rapidly becoming a key technology approach to addressing the issue of sample capacity. They increase sample capacity in two ways: 1) through accessing alternative, non-panel online sample sources, and 2) through more efficient use of respondents. This study is focused on the second way (sample efficiency). Routers promote sample efficiency through allowing respondents who disqualify from one study to attempt to qualify for other studies, thus enhancing the probability that they will be able to take a survey. Instead of discarding 50%+ willing respondents, we gain these respondents for immediate use on other studies. Respondent satisfaction is enhanced, and capacity for both a single study and for the overall system is increased. While routers have the potential to provide great benefits to the research industry, as with any sampling capability they need to be carefully managed and controlled. Routers are easy to create, but they are not easy to do well. The potential for bias in a poorly-designed router is tremendous. Although routers have been around since 1996, and are now being more widely used, relatively little research has been done to investigate the impact of router sampling management practices on survey results (using a single source; more research has been done on blending multiple sources). Some topics that have been explored to date include: priority/random assignment (Brigham, 2011a; Miller, 2010), selection bias due to reallocation (Brigham, 2011a; Porter, 2010), allowing respondents to take multiple surveys in one sitting (Johnson, 2010; Miller, 2010), panelist fatigue (Johnson, 2010), percent of sample reallocated in a single study (Brigham, 2011a), effect of screener position (Miller, 2010), and effect on bias of surveys at a lower priority (Brigham, 2011a). This study seeks to add to the existing body of knowledge by providing experimental evidence around three common router questions we hear from clients and suppliers. Brief Overview of Key Routing Concepts A router is a sampling technology that lets us allocate ( route ) respondents to surveys in real-time. It can be thought of as a group of surveys whose sampling is managed/controlled by a set of rules. Routers manage which Brigham & Fuller, 2012 CASRO Online Conference Page 2 of 14

3 Router Management: An Experimental Examination of Impact on Results surveys get presented to respondents. Respondents must qualify for a survey just as they do in nonrouter/traditionally-sampled studies. Router selection algorithm: Random vs. Priority Routers select studies for presentation to respondents based on randomness and/or a priority order. The router algorithm can range from 100% random (every survey has an equal chance of being presented to a given respondent), to 100% priority (each survey has a priority assigned to it, and the probability of it being presented is based on that priority), to any combination in between ( hybrid router ). For example, a 50% random/50% priority router would mean that 50% of the time a survey would be chosen randomly, and 50% of the time the survey chosen would be based on priority order of the survey. Although 100% random is ideal, it is often impractical in a business sense as certain studies need to be filled ahead of others. Most routers employ some form of hybridization. Reallocation In a non-router environment, sampling is typically done by sampling for respondents and sending them to a study according to that study s particular sample frame. We often use the term targeted respondents to refer to respondents who took the survey for which they were sampled and sent directly to. Reallocation is the process by which respondents who do not qualify for one survey can attempt to qualify for other surveys that are open to respondents at that time. Reallocation is source-agnostic ; it simply refers to how a certain respondent will be managed, no matter which source that respondent came from. Just because a respondent is reallocated to another survey, it does not necessarily mean that the respondent will take that survey. They must qualify, within quotas, for that survey. If they do not qualify, they may be reallocated to another survey, and so on, until they qualify for a survey. Screening process Versus a non-router/traditional approach, routers use an expanded screening process to qualify respondents. This is also known as a common screening process in that respondents may encounter screening questions for other surveys before they qualify for a certain survey. Therefore, the screening questions for one survey have the potential to impact other surveys either through affecting which respondents get into a survey, or affecting how survey questions are answered. This impact is often referred to in the context of correlation (e.g., how correlated two studies are). For example, if screening questions from Study A are highly correlated with Study B, it would mean that most respondents who qualify for Study A would also qualify for Study B. If Study A then also had a higher priority order than Study B, the respondent composition of Study B could be affected. The current research focuses on the impact of correlation as it affects which respondents get into a particular study. Key Research Questions Previous research by the authors and their Ipsos OTX colleagues, presented at the 2010 CASRO Online Conference and the 2011 CASRO Technology Conference, demonstrated how to measure selection bias due to reallocation Brigham & Fuller, 2012 CASRO Online Conference Page 3 of 14

4 Router Management: An Experimental Examination of Impact on Results and that this selection bias does not impact survey results in a diverse, moderately-sized router. These research conclusions then lead us (and our clients) to ask new questions how big and diverse does a router need to be? And what about the common screening process (where a respondent can potentially answer screening questions from multiple surveys before ultimately qualifying for and taking a particular survey)? Does it have an impact? These questions all center around a core concern: that one study is going to (negatively) impact another study. In a traditional, stand-alone approach to data collection, one study cannot impact another study, as they are completely separate studies. In a router, however, each study has the potential to impact, and be impacted by, other studies in the router. For this reason, we need to be concerned with the composition (types of studies), size of the router, and screening process. 1. Does spending more time in the router screening process have an impact on results? In a traditional stand-alone study, respondents answer only the screening questions for that particular study. All respondents who complete that study have answered the same questions, and same number of questions. In addition, each individual client can control how many screening questions are asked, and thus manage potential respondent fatigue. However, in the common screening environment of a router, respondents potentially answer multiple surveys screening questions before they qualify for and complete a survey. Therefore, respondents who complete a particular survey will likely have answered different screening questions and differing numbers of questions (and thus spent differing amounts of time in the screening process). 2. Does correlation among screening questions matter? Should a router be diverse in the types of studies it includes? It s a common assumption in the industry that routers should be diverse, but there is little experimental evidence for this supposition. One aspect of diversity in a router is correlation among screening questions. If the screening questions for a survey are highly correlated with other surveys screening questions, this may negatively impact who gets into a survey, and thus impact that survey s results. It is unrealistic to expect no correlation in a router, and indeed it is necessary for a successful respondent routing experience. But is there a limit to how much correlation can exist without biasing survey results? 3. How small can a router be are a minimum number of studies needed? One way to spread out the risk that one study will impact another is to have a router with many active studies. The conventional wisdom is the more studies in a given moment, the better. For most large routers, this is not an issue they have hundreds of studies in a router at any given time. However, there are some situations where routers may be much smaller (e.g., many router systems have mini-routers where a smaller number of studies are isolated). No experimental evidence has been presented to date on whether there should be a bottom limit. Is there a minimum number of studies, below which the risk for impact to survey results is greater? Brigham & Fuller, 2012 CASRO Online Conference Page 4 of 14

5 Router Management: An Experimental Examination of Impact on Results Research Design & Analysis In order to test multiple router management practices in a robust, statistically valid way, we first created an experimental design that would provide the flexibility to test multiple practices using both direct comparisons and simulations, and then created a router environment whose operational variables we could control. The experimental design In order to isolate router effects (vs. sample source effects), the experiment involved only one sample source (a panel). Four surveys were included in the study (all concept tests). Each survey was fielded in both a router and a non-router (traditional, stand-alone) environment. The non-router environment provided a control for the direct comparisons. Control cells o We pulled sample according to each survey s sample frame and sent panelists directly to the survey for which they were sampled. o Respondents answered the screening questions for that survey and were matched against quotas. o Non-qualifiers skipped the actual survey and answered a battery of screening questions (from phantom router surveys, explained below). Respondents who took the actual survey also answered these screening questions at the end of the survey. Test (router) cells o We created a router environment containing 13 studies. Four of the 13 studies were the test surveys; the other 9 were phantom studies. For the phantom studies, respondents answered the studies specific screening questions but did not actually take the phantom surveys. o We pulled samples according to all 13 surveys sample frames, and these samples were sent into the router. o Respondents were first asked a number of screening questions. They were then randomly sent to one of the four test surveys. All respondents, including those pulled for the phantom studies, were sent to a test survey and completed that survey if they qualified for it. (Depending on which studies were active/live in a simulation, and how quotas filled up for each study in the simulation, the phantom study respondents would be considered reallocated respondents.) o Non-qualifiers skipped the test survey and answered the remaining screening questions (all respondents answered all screening questions). Respondents who qualified for and completed the survey answered the remaining screening questions at the end of the test survey. o The usual field quotas were monitored for all 13 studies (including over-quotas), and boost samples for each of the 13 studies were pulled as needed to fill quotas. In total, over 10,000 respondents completed the Test surveys. Each Control survey had completions. Brigham & Fuller, 2012 CASRO Online Conference Page 5 of 14

6 Router Management: An Experimental Examination of Impact on Results Creating the router environment In order to have control over the operational variables of the router, we had to create our own router environment. However, we wanted this environment to be as realistic as possible, mirroring the typical composition of studies we would see in a router at a large full-service research firm. 1 o To select the 9 phantom studies, we analyzed 12 months of Ipsos online business and randomly selected 9 studies that proportionally represented the types of studies we do. o The 13 studies had diverse sample frames. For Test 3 (used in the simulation analyses), bivariate correlations among screening questions ranged from -.15 to Correlations were computed through comparing qualification (at respondent level) for 3 to each of the other surveys (see Figure 1). For example, 6 s correlation of.93 with 3 meant that most of the respondents who qualified for 6 also qualified for 3 (the Test survey). o Sample frames, screening questions, and quotas were managed separately for all 13 studies. Samples were pulled to match each survey s usual sample frame. All usual screening questions were asked for each survey. Field quotas were monitored for each survey separately; if boost samples were needed to fill quotas, those boosts were pulled according to the survey s specific sample frame. 2 3 (TEST) Figure 1. Bivariate correlations among Test survey and other surveys screening questions. Analysis Both direct comparisons and simulations were employed to test the research questions. For the direct comparisons, we compared Test results to the corresponding Control cells. The simulations used the Test results only; the control in these analyses came from a simulation that mimicked a non-router (stand-alone) environment. We performed the direct comparisons on all four test surveys. For the simulations, due to the time needed to run each simulation, we analyzed results from one survey. We chose three key concept test metrics to analyze. For the simulations, we created scenarios that mirrored different router management practices. Each scenario was tested using a 50% random/50% priority router algorithm (priority order for studies was assigned randomly at the start of a group of simulations), with 50% targeted/50% reallocated sample allowed in each study. We used respondents actual survey, screening, and quota information in the simulations. The survey s specific quotas were applied in each simulation, and we let quotas fill up as usual. Each simulation had 350 respondents, and we ran 100 simulations per scenario. We then averaged the results for each of the 3 key metrics across all 100 simulations. For a control, we ran multiple simulations on a scenario of a stand-alone study with no reallocation. 13 Brigham & Fuller, 2012 CASRO Online Conference Page 6 of 14

7 Router Management: An Experimental Examination of Impact on Results Research Results Although with this research design we cannot completely isolate the impact of the screening process (from routing/reallocation bias), we can rule out major contributing factors and draw general conclusions on the impact. Unpublished research by the authors (Brigham, 2011b) has demonstrated that there is no significant difference in survey results between targeted (traditional sampling approach) and reallocated respondents in a router environment. In addition, previous research presented at CASRO (Brigham, 2011a) demonstrated that routing bias (under the same conditions as the current research) does not have a significant impact on survey results. Therefore, comparing a non-router/traditional approach to a router approach allows us to draw general conclusions about the impact of the expanded router screening process on survey results. For this paper, we examined impact of time spent in the router screening process in two ways: 1) overall comparison of a router approach to a traditional approach, and 2) for router respondents, comparing those who spent more time in the process to those who spent less time. It should be noted that, due to necessities of the experimental design, router test respondents answered an inflated number of questions vs. what they likely would have in a typical router situation. However, this provides a more liberal estimate of potential impact. Increased router screening did not impact survey results Even with the inflated increase in screening for the router respondents, no systematic or significant differences were found for the key metrics between the Test and Control groups. Figure 2 shows all 12 comparisons (3 metrics for each of the 4 surveys). Figure 2. Comparison of Router (Test) and Traditional/Non-router (Control) key metrics. Brigham & Fuller, 2012 CASRO Online Conference Page 7 of 14

8 Router Management: An Experimental Examination of Impact on Results Spending a greater amount of time in the router was not related to the screening process We first looked at how much time was actually spent by respondents on average. As would be found in a typical large router, our screening questions encompassed both easy to answer questions (e.g., age, gender) and questions that required more thought and processing (e.g., complex category screeners). For example, one of the test surveys had a category screening process that required seven complex questions (and all respondents were required to answer these questions before proceeding to a survey). As expected, respondents on average did not spend a long time in the expanded router screening process. Respondents spent an average of 6 seconds a question (although this is likely inflated due to the required complex test survey question referenced above we estimate a more realistic average would be around 5 seconds a question). The median time spent was 1 minute, and only 5% of respondents spent more than four minutes. Figure 3 illustrates this. Figure 3. Time spent in router screening process (average across respondents). We next examined how many questions respondents answered on average, by the time they spent in the router. In order to compare respondents by time, we grouped respondents by each minute spent (1, 2, 3, and 4+ minutes). Some respondents did spent less than 1 minute in the process, but as the base size was small we did not include them in the analysis. Brigham & Fuller, 2012 CASRO Online Conference Page 8 of 14

9 Router Management: An Experimental Examination of Impact on Results Figure 4 shows that respondents in the 2, 3, and 4+ minute groups answered the same number of questions on average. Therefore, spending additional time in the router does not appear to be related to the router screening process. Figure 4. Average number of questions answered in screening process, by time spent. A highly correlated router produced skewed results; including a mixture of correlations brought results back into line To analyze potential impacts of correlation, we created simulation scenarios that mimicked different amounts of correlation among screening questions. We created two boundary conditions High Correlation where all studies were highly correlated with the Test concept (average correlation of.69) and Low Correlation where all studies were uncorrelated with the Test concept (average of.004). (While the Low Correlation router is not realistic, it is the closest configuration to a stand-alone environment, and thus serves as a good control to benchmark against.) We also created an Average Correlation condition, with a mixture of high, medium, and low correlations (average of.32). The High Correlation router skewed higher, producing higher values for two of the three metrics (the third metric was flat, so it is not shown*). However, including a mixture of correlations, as in the Average Correlation router, brought results back in line with the Low Correlation router. It should be noted that, although these results were within the standard deviations for each metric, the trend is clearly there. Figures 5a & b illustrates these patterns. * The third metric was value-related and did not show a great deal of deviation in survey responses. While we included it in the direct comparisons, we did not feel that the router simulation results would necessarily reflect a truthful impact, due to the flatness of the responses. Therefore, we did not show these results. Brigham & Fuller, 2012 CASRO Online Conference Page 9 of 14

10 Router Management: An Experimental Examination of Impact on Results Figures 5a & b. Comparisons of key metrics for different router configurations of correlation among screening questions. Metric 1: avg SD=.09, Metric 2: avg SD=.08 We must caveat these correlation findings at this point, however. While we show a modest impact of correlated screening questions, the actual impact may be much higher in other router configurations. Biases created by the router environment will be a function of a number of conditions: prioritization of a study, prioritization algorithm of the router, number of studies active in the router, etc. Our simulations were run under conditions that guard against/mitigate some of the potentially negative impacts: router prioritization was 50% random, and study prioritization was random assignment. In more extreme configurations, such as 100% prioritization, the impact shown could theoretically be much higher. More studies are better Test metrics are closer to Control around surveys in a router To address the question of minimum size, we ran scenarios with differing numbers of active studies (1, 2, 4, 6, 8, 10, and 12 studies). For example, the 2-study scenario was simulated 100 times; in addition to the Test survey, one other study was active/live in the router. The one-study scenario can be considered a pseudo-control although only one study is active, it differs from a non-router stand-alone study in that it has access to all sample that enters the router (in this case, sample that was targeted toward the other 12 studies). For a traditional control, the study would only have access to the specific sample that was targeted toward and sent directly to it. We compared results on the key metrics for all scenarios (including the pseudo-control) to a traditional, standalone control. We looked at the patterns of results to assess if we see different results when there are fewer Brigham & Fuller, 2012 CASRO Online Conference Page 10 of 14

11 Router Management: An Experimental Examination of Impact on Results studies in the router. We also examined whether there is a tipping point where we see results line up with the control. Figures 6a & b show patterns for two of the metrics (metric 3 was again flat). Relative to the stand-alone control, as we move studies into the router and onto reallocation, the metric values increase. With only one study active, the increase is highest. As more active studies are added, the values start to decline, stabilize, and move toward the control. While we didn t observe a clear tipping point, metric values appeared to move closer to the control between surveys. Again, it should be noted that, while results were within standard deviations for each metric, the trend is clearly there. Figures 6a & b. Patterns of results for different numbers of studies active in a router. Metric 1: avg SD=.09, Metric 2: avg SD=.08 Conclusions / Implications The nature of inter-relationships among studies in a routing environment presents a concern for researchers: that one study will impact another study, often negatively. While this is a valid concern, understanding how to appropriately design, manage, and control a routing environment can minimize any potential impact. The current research sought to investigate the impact of router management practices, and provide evidence to address concerns in this area. The common screening process is a key area of concern, given that a respondent may encounter screening questions for multiple surveys before they complete a survey. Key issues involve the impact of the expanded screening process (adding to what is often an already long survey process), and the impact of correlated screening questions (reflecting that respondents may qualify for multiple studies). Brigham & Fuller, 2012 CASRO Online Conference Page 11 of 14

12 Router Management: An Experimental Examination of Impact on Results Our research found that although respondents spent differing amounts of time in the screening process, after 1 minute, time spent did not appear to be related to the screening process itself. Respondents who spent 2, 3, and 4+ minutes in the screening process answered the same number of questions on average. In addition, the extra questions did not add a great deal of time to the screening process; even with them, the median time spent in the screening process was 1 to just under 2 minutes. It is worth noting that our study was a more extreme scenario given that every respondent had to answer at least 9 screening questions, and some answered up to 22. This is likely not the case in a typical router, where some respondents answer no additional screening questions. Another area of concern in a router (related to the common screening process) is that of diversity among studies. Two studies are correlated to some degree in their screening questions, and this will impact which respondents qualify for those studies. If one study is highly correlated with another, than there is potential for it to obtain respondents at the expense of the other. In many cases, this is likely not as much of a concern. However, in some cases this can skew who gets into a study. For example, suppose there is both a study on smoking and one on teeth whiteners. Arguably, smokers would be more likely to use teeth whiteners than other groups in the population, and this would need to be reflected in the respondent composition for the teeth whitener study. However, if the smoker study had a higher priority than the teeth whitener study, and most of the smokers qualified for the smoker study, fewer smokers than needed might qualify for the teeth whitener study. This could bias the teeth whitener survey results. Our results indicate that a router with a diverse mixture of correlations (low, moderate, and high) among screening questions performs closest to what we would view as a control a router where all studies are very lowly-correlated (closest configuration to a stand-alone, non-router environment). While this is good news for larger routers which tend to have diverse types of studies, it suggests that extra care should be taken with smaller routers. In addition, this research only examined one type of correlation. Another type of correlation among screening questions is when a study s screening questions are correlated with the key metric of another study (such as Purchase Intent). This is arguably a more dangerous type of correlation as it is harder to see, and thus control. Future research should examine the impact of that type of correlation. Diversity of studies in a router is often related to the size of the router. While we have heard questions from clients on whether the size of a router should be capped, we feel the actual question should be how small is too small? Are there a minimum number of studies needed in order to minimize impact to results? (It should be noted that having fewer studies in a router also risks issues that we did not cover in our analyses for example, highly targeted studies producing respondent skews.) Our data indicates that at least 10 active studies are needed in a router to minimize impact to results. We view this as the bare minimum, and recommend more research be done to investigate patterns of results when there are 13 to 20 active studies. Our research shows that vs. a control, results were impacted by having fewer studies active, and results seemed to move closer to the control around studies. However, while the trend was there, it was not strong and simulations with between surveys should be done to ascertain whether this is a stable result. Brigham & Fuller, 2012 CASRO Online Conference Page 12 of 14

13 Router Management: An Experimental Examination of Impact on Results These insights are promising, and we plan to analyze the other 3 test surveys to confirm findings from this test. In summary, routers bring many benefits to our industry, but just as with any data collection/sampling method they need to have appropriate router management practices in place. Our research indicates that diversity and size of a router are important. Additional time spent in the screening process does not appear to be as much of a concern as anticipated (but this result should not be construed as an open invitation to add screening questions to the survey process!). We would also like to emphasize that these results are valid for a typical router that employs controls such as hybrid prioritization. They may not be valid for routers that are managed differently, and future research needs to be done to understand them further. However, the results should offer some relief to researchers and clients in a large, diverse, appropriately-managed router, one study negatively impacting another study should not be a major concern. Brigham & Fuller, 2012 CASRO Online Conference Page 13 of 14

14 Router Management: An Experimental Examination of Impact on Results References Brigham, Nancy, Scott Porter, Lee Markowitz, & Jason Fuller (2011a). Sampling with Routers: Comparing Results of Reallocated and Traditionally Sampled Respondents. Presentation given at the 2011 CASRO Technology Conference. Brigham, Nancy, Scott Porter, Lee Markowitz, & Jason Fuller (2011b). Unpublished research on targeted vs. reallocated respondents in a router environment. Johnson, Paul, & Bob Fawson (2010). Factorial Design on Router Effects. Paper given at the 2010 CASRO Panel Conference. Miller, Chuck (2010). Route 66: The Long Road to Efficient and Effective Routers. Presentation given at the 2010 CASRO Technology Conference. Porter, Scott, Olivier de Gaudemar, & Mark Kimura (2010). Measuring Selection Bias Introduced by Routing. Paper given at the 2010 CASRO Panel Conference. Biographies Nancy Brigham, Ph.D. Vice President, Global Operations, Respondent Access & Engagement Ipsos Interactive Services As an expert in research strategy and business application, Nancy Brigham has worked across both the client and supplier sides of market research. At Ipsos, Dr. Brigham leads various respondent access and engagement initiatives across the Americas, including research-on-research for Ipsos router sampling technology. Prior to joining Ipsos, she spent 12 years at P&G, where she provided advanced methodological/analytical understanding and business application in diverse research areas, including upstream capability development, consumer insights, quality, and operations. She holds Doctorate and Masters degrees in Social Psychology, with concentrations in Statistics and Methodologies, and a Bachelor s degree in Advertising and Psychology. Jason Fuller Director, Panel Analytics Ipsos Interactive Services Jason has worked at Ipsos (formerly NPD) for 11 years. Jason is responsible for developing best practices and providing research-on-research for the Online Respondent Access & Engagement Services group. During his tenure at Ipsos, Jason has managed and developed the offline panels for Ipsos North America. He has also designed and managed a variety of longitudinal and custom projects for the North American Panel group. Jason holds a Bachelor s degree, with honors, in Statistics. Brigham & Fuller, 2012 CASRO Online Conference Page 14 of 14