Introduction to Questionnaire Design Why this seminar is necessary! Bad questions are everywhere! Don t let them happen to you! Fall 2012 Seminar Series University of Illinois www.srl.uic.edu The first presidential debate was last night. Did you watch it and will it affect your vote? Yes, I m still undecided No, I already know my candidate I m not going to vote 1 2 When Bad Questions Happen to Good People I am usually pretty calm and relaxed. How accurate is this statement as a description of you? Basic Steps Specify constructs you want to measure(think about your analysis plan ahead of time!) Look for existing questions before drafting new ones (search GSS SUBJECT INDEX ) Place your questions in logical order and set up skip patterns Have your questionnaire reviewed Pretest! 3 4 Cognitive Steps in Answering Questions 1. Understand question. 2. Search memory for information. 3. Integrateinformation into summary judgment. 4. Translatejudgment onto response alternatives. Types of Measures Those that ask about behaviors or facts Those that ask about psychological states or attitudes Those that ask about knowledge 5 6
7 Ask about one question at a time. Double-barreled: Compared to one year ago, are you paying more, less, or about the same for your auto and life insurance? Compared to one year ago, are you now paying more, less, or about the same for a) Auto insurance b) Life insurance Use specific, simple language. Complex question: What percentage of your patient population has proficiency in English? Approximately what percent of the patients in your practice are unable to speak English? 8 Minimize social desirability Did you vote in the general election on November 6, 2012? Sometimes, despite intentions, people are not able to vote on election day. What about you? Did you vote in the general election on November 6, 2012, or did you not vote? Avoid yes/no, true/false, and agree/disagree questions! Choose questions with construct-specific response options instead for better data quality 9 10 Agree-Disagree Example Typical question: How much do you agree or disagree with the following statement: The Dean promotes diversity within the college. Would you say you strongly agree, somewhat agree, somewhat disagree, or strongly disagree. Improved Question that is Construct Specific How effective is the Dean in promoting diversity within the college? Would you say extremely effective, very effective, moderately effective, slightly effective, or not at all effective? 11 12
13 Write questions with balanced wording to reduce bias. In a typical day, do you brush your teeth two or more times a day? Becomes In a typical day, do you brush your teeth two or more times a day, or do you not brush your teeth two or more times a day? Specify who, what, when, where and how. In 2011, what was your total household income, before taxes? Please count income from all membersof your household, includingwages from employment, disability, social security, and public aid. 14 Specify through cues: People drink beer in many places for example, at home, at restaurants, at bars, sporting events, and at homes of friends and family. During the past 30 days, did you drink any beer? Use words with singular meanings: Ambiguous: How would you compare how close you are to family in your current neighborhood to how close you were in your old neighborhood? Clear: Compared to your last neighborhood, do you now live closer to your family, are you further from your family, or are you about the same distance? 15 16 Open vs. Closed Questions Closed questions are usually better Easier for the respondent, and R preference Less coding later = less expensive Better to have respondent do categorizing Instead of how many years have you been a student at UIC? Less than 1, 1-3, more than 3 just leave a line and let them write in response. Disadvantages of Closed Questions Categories may be leading to respondents More work for the R: need to answer question in your mind first, and then look for answer category that fits you May limit spontaneity Not best when asking for frequency of sensitive behaviors; response alternatives suggest range of normalcy to the R there are numerous possible responses role of initial question development and pretesting 17 18
19 Disadvantages of Open-ended ended Questions With interviewer administered questionnaires, extensive training is required for proper probing techniques Elaborate coding schemes Expensive and time consuming (to reduce and analyze) Only ask if you really intend to use the data Question Order Segment questions by topic (ask about related topics together) Ask recall backwards in time Use transitions when changing topics Leave objectionable questions for the end Always ask demographic questions at the end, unless you need them for skip branches 20 Response Categories Based on your experiences (and what your children have shared so far) what are your favorite elementsof the program? Unsatisfactory Good Excellent Don t Know Chess Legos Drama Improv Circus Arts Response Categories Response categories should be consistent with the question Response categories should cover only one dimension of response Categories should typically be exhaustive, including every possible answer Categories should be mutually exclusive (no overlap) 21 22 Response categories should be consistent with the question: When you eat at home, do you prepare the food yourself? Always Most of the time Sometimes Rarely Never When you eat at home, how often to you prepare the food yourself? 23 24
25 To what extent does your department use annual performance reviews to enhance staff development? Check all that apply. Another common mistake: response categories encompass more than one dimension. For example: The to what extent wording suggests that respondents will be asked to rate each item on a scale from not at all to a complete extent, but the format was check all that apply. How much influence do the following institutions exert over your department? Very Strong Influence Strong Influence Moderate influence Mild influence No influence 26 Response Category Scales Suggested response scale that measures the question construct: How much influence do the following institutions exert over your department? No influence at all A little influence Some influence A lot of influence A great deal of influence Unipolar vs. Bipolar Response Scales: Unipolar: Not at all Extremely ( 5 points) Bipolar: Neutral in the middle (7 points) (Hate/Neutral/Love) 27 28 Unipolar vs. Bipolar Unipolar vs. Bipolar In general, how easy or difficult is it to retain qualified staff? Very easy Easy Neither easy or difficult Difficult Very difficult How difficult is it to retain qualified staff? Not at all difficult Slightly difficult Moderately difficult Very difficult Extremely difficult 29 30
31 Examples of Good Unipolar Response Scales All 5 point response scales: Extremely, Very, Moderately, Slightly, Not at all A great deal, a lot, a moderate amount, a little, not at all Definitely will, Probably will, Might or might not, Probably won t, Definitely won t Always, most of the time, about half of the time, sometimes, never Excellent, Good, Fair, Poor, Very Poor Examples of Good Bipolar Response Scales 7-point response scales: Extremely good, moderately good, slightly good, neither good or bad, slightly bad, moderately bad, extremely bad Like a great deal, like a moderate amount, like a little, neither like or dislike, dislike a little, dislike a moderate amount, dislike a great deal 32 Response Category Issues Data show that the longer the response scale, the longer it takes Rs to put themselves on it, except in the region of 5-7 points. Reliability and validity are also strongest when unipolar response scales are 5 points and bipolar scales, 7 points. Factors that lead R s to pick don t know vs. a middle category are not interchangeable Midpoint responses in bipolar scales are not disguised don t know responses; if it s possible your R won t know, leave them that option. Physical Format Checklist Number all questions sequentially Use large, clear type; don t crowd White space place more blank space between questions than between subcomponents of questions List answer categories vertically instead of horizontally Avoid double/triple banking of response choices 33 34 Physical Format Checklist (cont d) Don t split questions across pages. Put special instructions on questionnaire as needed, next to question Distinguish directions from questions be consistent Be consistent with direction of response categories Be consistent with placement of response categories Introduction/Cover Letters Introduction should indicate: who is conducting the survey the topics to be covered in the survey an assurance of confidentiality any IRB stipulations whether you offer how long it will take depends on mode, topic, population 35 36
37 Pretest! Never skip a pretest Only need a small sample of respondents, preferably representing all groups in population of interest Hold debriefing with interviewers to get their feedback For self-administered surveys, can also debrief with subjects to get feedback