The Challenges Associated with Relying on CAPI Interviewers to Implement Novel Field Procedures

Transcription

1 The Challenges Associated with Relying on CAPI Interviewers to Implement Novel Field Procedures Gina Walejko, U.S. Census Bureau James Wagner, University of Michigan American Association for Public Opinion Research May 15, 2015 Any views expressed on (statistical, methodological, technical, or operational) issues are those of the author(s) and not necessarily those of the U.S. Census Bureau. 1

2 Problem Data Compliance Metrics Results Conclusion Outline 2

3 Implementing Adaptive Designs Intervention requires adherence to the protocol. Examples of CATI protocols Prioritized/targeted cases (Laflamme & St-Jean, 2011; Peytchev et al., 2010) Assigned cases to call windows based on estimated contact propensity (Wagner, 2013a) Assigned best interviewers to cases w/ lowest cooperation (Luiten & Schouten, 2013) Put overrepresented cases on hold (Coffey, 2013; Lundquist & Särndal, 2013) Examples of CAPI protocols Prioritized cases (Wagner et al. 2012; Walejko et al. 2013) Recommended call times based on estimated contact propensity (Wagner, 2013b) 3

4 Interviewer Compliance Research shows interviewers do not: Read questions as worded when trained to do so (Cannell et al., 1968; Cannell & Oksenberg, 1988; Fowler & Mangione, 1986; Ongena & Dijkstra, 2000) Accurately record contact attempts as trained (Biemer et al., 2013; Laflamme & Karaganis, 2010) Lack of interviewer compliance can: 1. Lead to failure to implement intervention 2. Make it difficult to analyze effects of an intervention 3. Affect analyses that rely on interviewer-collected data 4

5 2013 & 2014 Census Tests 2013 Census Test 2014 Census Test Adaptive Fixed Fixed Control Contact Contact Adaptive Cases in Field ,642 12,466 8,476 Interviewers Site Philadelphia Washington DC area Contact History Separate instrument Built into survey instrument Training 2 + ½ day refresher 3 days 5

6 2013 Compliance Metrics 1. Work prioritized cases daily 2. Transmit data twice daily Screenshot of 2013 Census Test Case Management System Adaptive Condition 6

7 2014 Compliance Metrics 1.Work proxy cases when instructed 2. Perform telephone calls when instructed Screenshots of 2014 Census Test Case Management System Adaptive & Fixed Conditions 7

8 Research Questions Do interviewers transmit twice daily? Do interviewers attempt all priority cases each day they work? Do interviewers perform proxy attempts when instructed? Do interviewer perform telephone calls when instructed? Why are interviewers not compliant? 8

9 Transmit Data % i = % 21-40% 41-60% 61-80% % Interviewers (Fixed & Adaptive) Grouped by Percent Compliant Transmissions (n=18) 9

10 2013 Work Priority Cases Daily Percent Days with the Following Calling Patterns Among Days Interviewers (Adaptive) Who Transmitted Correctly All High Priority Cases Compliant 45% Some High Priority Cases, No Regular Cases Some or No High Priority Cases, Some Regular Cases Possibly Compliant 7% Not Compliant 48% 10

11 Perform Proxy Attempts Number and Completion Rate of Compliant and Non-Compliant Proxy Attempts (Adaptive Interviewers) n Compliant Percent Compliant Percent Completed n Not Compliant Percent Percent Compliant Completed % 20% % 20% Note: Excludes cases not attempted and those found to be vacant or not housing units. 11

12 2014 Mode of Contact (1) Number and Percent of Compliant and Non-Compliant Mode Contact Attempts by Contact Attempt Number (Fixed Interviewers) Compliant Not Compliant Contact Attempt n Percent n Percent 1 8, , , , , Total 25, , Note: Excludes records of inbound telephone call attempts. 12

13 2014 Mode of Contact (2) Compliant Not Compliant Completion Rates by Contact Attempt Number for Compliant and Non-Compliant Mode Attempts (Fixed Interviewers) 13

14 Summary 2013 Census Test interviewers: Transmitted data as instructed 71% of days. Attempted all priority cases on less than half of days worked and transmitted correctly Census Test interviewers: Performed proxy attempts 83% of time when instructed. Performed compliant mode contacts over 88% of days worked. 14

15 Conclusions Interviewer compliance is essential to implementing fieldwork innovations, such as in adaptive design. In face-to-face surveys, interviewer compliance is uncertain. Training, by itself, does not solve the issue. Understanding why non-compliance occurs is important may lead to solutions. Systems may provide solutions: User involvement in design of decision support improves compliance (Kawamoto, et al., 2005) Deliver next case as is done in CATI facilities? Build infrastructure to incentive interviewers in following protocol? 15

16 Questions? 16

17 Citations (1) Biemer, P., Chen, P., & Wang, K. (2013). "Using level-of-effort paradata in non-response adjustments with application to field surveys." Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): Cannell, C., Fowler, F., & Marquis, K The influence of interviewer and respondent psychological and behavioral variables on the reporting in household interviews, Vital and Health Statistics, Series 2(26). Washington, DC: U.S. Government Printing Office. Cannell, C. & Oksenberg, L. (1988). Observation of behavior in telephone interviewers. Pp in Telephone Survey Methodology, edited by R. Groves et al. New York: Wiley. Coffey, S. (2013). Implementing Adaptive Design for the National Survey of College Graduates. FedCASIC. Fowler, F. & Mangione, T. (1986). Reducing Interviewer Effects on Health Survey Data. Washington, DC: National Center for Health Services Research. Kawamoto, K., Houlihan, A., Balas, E., & Lobach, D. (2005). Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. British Medical Journal 330(7494): Laflamme, F. & Karaganis, M. (2010). Assessing Quality of Paradata to Better Understand the Data Collection Process for CAPI Social Surveys. European Quality Conference, Helsinki, Finland. Laflamme, F. & St-Jean, H. (2011). Highlights from the First Two Pilots of Responsive Collection Design for CATI Surveys. Joint Statistical Meetings. 17

18 Citations (2) Luiten, A. & Schouten, B. (2013). Tailored fieldwork design to increase representative household survey response: an experiment in the Survey of Consumer Satisfaction. Journal of the Royal Statistical Society, Series A, 176(1), Lundquist, P. & Särndal, C.-E. (2013). "Aspects of responsive design with applications to the Swedish Living Conditions Survey." Journal of Official Statistics 29(4): Ongena, Y. & Dijkstra, W. (2006). "Methods of behavior coding of survey interviews." Journal of Official Statistics 22(3): 419. Peytchev, A., Riley, S., Rosen, J., Murphy, J. & Lindblad, M. (2010). "Reduction of nonresponse bias in surveys through case prioritization." Survey Research Methods 4(1): Wagner, J. (2013a). Adaptive contact strategies in telephone and face-to-face surveys. Survey Research Methods, 7(1), Wagner, J. (2013b). Using paradata-driven models to improve contact rates. In Kreuter, F. (Ed.), Improving Surveys with Paradata: Analytic Uses of Process Information. Hoboken, New Jersey: John Wiley and Sons. Wagner, J., West, B., Kirgis, N., Lepkowski, J., Axinn, W., & Ndiaye, S. (2012). Use of paradata in a responsive design framework to manage a field data collection. Journal of Official Statistics, 28(4), Walejko, G., Keller, A., Dusch, G., & Miller, P. (2014) Research and Testing: 2013 Census Test assessment. Washington, DC: U.S. Census Bureau. 18

19 Additional Slides 19

20 Transmit Data 2014 Census Test Compliant Daily Transmissions by Treatment Treatment Interviewers Total Days Worked Frequency Compliant Transmissions % Compliant Transmissions Control Experimental Adaptive Total

21 Transmit Data % Enumerators Compliant Per Day % Interviewers Compliant /13 8/15 8/17 8/19 8/21 8/23 8/25 8/27 8/29 8/31 9/2 9/4 9/6 9/8 9/10 9/12 9/14 9/16 9/18 Date of Field Period Percent Daily Compliant Transmissions for Interviewers who Worked During 2014 Census Test 21