School Concepts for Spanish Speaker Respondents Improving concept validity in surveys through cognitive interviews. Anna Sandoval Girón Researcher American Institutes for Research, Center for Survey Methods American Institutes for Research Established in 1946, with headquarters in Washington, D.C., American Institutes for Research (AIR) is an independent, nonpartisan, not-forprofit organization that conducts behavioral and social science research and delivers technical assistance both domestically and internationally. As one of the largest behavioral and social science research organizations in the world, AIR is committed to empowering communities and institutions with innovative solutions to the most critical challenges in education, health, workforce, and international development. May 2014 Copyright 2014 American Institutes for Research. All rights reserved. 2 Outline 1. Background information previous work on translation Theoretical background 2. The research 3. Findings Theoretical background Cognitive interviewing for survey design. What we are missing is empirical knowledge about how use of interpreters or use of monolingual questionnaires in bilingual settings affects data quality (Carrasco, 2003). 3 4 1
Assumptions Response error is the most common source of survey error. Question interpretation by the respondent may lead to response error. Cognitive interviewing has been successful as a tool for questionnaire development to explore questionnaire improvement. Respondents whose native language is not the same as the source language may encounter difficulties. Research Project - NHES Two iterative rounds of cognitive testing on a series of the National Household Education Surveys (NHES) educational questions with monolingual Spanish-speaking respondents from different countries living in the United States. Interviews were conduct in San Mateo, CA (bay area) and the DC metro area. Interviews were conducted in September of 2013. Protocols were testing two areas, The translation concepts Instructions for the survey 5 6 Concepts tested homeschooling charter schools after-school programs Homeschooling Background Initial translation NHES 2012 Census translation (ACS version for Puerto Rico) Adaptation of census translation NHES 2013 Homeschool NHES definition: Homeschooled students are school-age children (ages 5 17) in a grade equivalent to at least kindergarten and not higher than 12th grade who receive instruction at home instead of at a public or private school either all or most of the time. 7 8 2
Homeschool - continued NHES English Version 2012 homeschool instead of school for some or all classes NHES Spanish version used in 2012 Recibe algunas o todas sus clases en el hogar en vez de en una escuela. Takes some or all classes in the home instead of at a school. Homeschool - continued ACS census version (for Puerto Rico) Fue esa una escuela o universidad pública, una escuela o universidad privada o era enseñanza en el hogar (home school)? Was that a public school or college, a private school or college, or home school (home school)? NHES Spanish version tested in 2013 Educación en el hogar (homeschool) en vez de una escuela para algunas o todas las materias. Homeschool instead of school for some or all classes. 9 10 Findings Five of the 8 respondents had trouble with the question. Three of the five respondents understood the term homeschooling to mean that the child takes some instruction at home by a teacher. For example: Recibe clases en casa Takes classes at home Homeschooling - continued Issues: No linguist equivalent to homeschooling No cultural equivalent to homeschooling Attempts to fix the problem have had mixed results: 3 of the 8 respondents recognized that their child was not a part of homeschooling and answered no to the question. Testing will continue with another round of cognitive interviews this summer. Uncertainty over whom provides the teaching at home. 11 12 3
Homeschool current version Charter school Question: Es esta escuela una escuela Charter? Is this school a charter school? This is a direct translation. There is no Spanish equivalent for Charter school. 13 14 Results After-school programs During probing four out of the eight respondents stated they didn t know what a charter school is. One respondent answered the question correctly (her child attends a public school). Three respondents skipped the question. Is this school a charter school? 15 16 4
Overall Findings Translations of educational level items require pre-testing in numerous iterations. Conceptual translations may lead to more complex (wordy) translations which reduces the ability of low literacy respondents to accurately answer questions. References Carrasco, Lorena. (2003). The American Community Survey (ACS) en Español: Results of Cognitive Interviews Using the ACS Spanish Language Instrument. Paper presented at the Federal Committee on Statistical Methodology (FCSM) Washington, DC. Fernández, Leticia, Goerman, Patricia, & Quiroz, Rosanna. (2012). Translation of Survey Items on Country Specific Programs: The Case of Translating U.S. Educational Level Questions into Spanish Washington, D.C.: U.S. Census Bureau. Mackey, William F. ([1987] 2005). The Description of Bilingualism. In B. Mayor & A. K. Pugh (Eds.), Language, Communication and Education (pp. 291-307). Wolfebore, New Hampshire: Taylor and Francis. Noel, Amber, Stark, Patrick, Redford, Jeremy, & Zukerberg, Andrew. (2013). Parent and Family Involvement in Education, from the National Household Education Surveys Program of 2012 (N. C. f. E. Statistics, Trans.). Washington, DC: National Center for Education Statistics. Tourangeau, Roger, Rips, Lance J, & Rasinski, Kenneth. (2000). The Psychology Of Survey Response. New York: Cambridge University Press. 17 18 Anna Sandoval Girón 202-403-5389 asandovalgiron@air.org 1000 Thomas Jefferson Street NW Washington, DC 20007 General Information: 202-403-5000 TTY: 887-334-3499 www.air.org www.csmair.org Screener Version (English) homeschool instead of school for some or all classes Spanish version used last year (2012) Recibe algunas o todas sus clases en el hogar en vez de en una escuela. Spanish version tested this year (2013) Educación en el hogar (homeschool) en vez de una escuela para algunas o todas las materias. Version for the screener Participa en el sistema de padres educadores en el hogar (homeschool) en vez de asistir a la escuela para algunas o todas las clases. 19 20 5