Questionnaire Design Clinic

Transcription

1 Questionnaire Design Clinic Spring 2015 Seminar Series University of Illinois 1 Cognitive Steps in Answering Questions 1. Understand question. 2. Search memory for information. 3. Integrate information into summary judgment. 4. Translate judgment onto response alternatives. 2 1

2 Survey Intro/Cover Letter Introduction should indicate: who is conducting the survey the topics to be covered in the survey an assurance of confidentiality any IRB stipulations whether you offer how long it will take depends on mode, topic, population 3 Survey Intro Telephone or Inperson example Hi, my name is. I m (calling) from the University of Illinois Survey Research Laboratory. We re working with the Department of to conduct a survey to understand the patterns of interactions between the groups and individuals who will be attending.. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the region. Your participation is voluntary and you may choose not to answer any questions if you do not want to just let me know. [GO INTO FIRST QUESTION] Tailored by interviewers: Not selling anything First 30 seconds (particularly on the telephone) 4 2

3 Cover letter - example The Department of at the University of Illinois at Chicago, is conducting to understand the patterns of interactions between the groups and individuals. The research team is headed by You are receiving this questionnaire because you are... It should take you about 10 minutes to complete the questionnaire. Please respond by August 1. Participation in this survey is voluntary, and you may choose to stop answering questions at any time. Your choice to participate or not will have no effect on your course grade or on your relation with the University of Illinois. Your response is anonymous as we will not be asking for any personally identifying information please do not write your name anywhere on this questionnaire. We plan to use the results of this survey in part for University of Illinois at Chicago research purposes. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the Calumet region.. Only aggregated results will be presented. Completing this questionnaire indicates your consent to participate in the survey. Please contact if you have any questions. If you have any questions about your rights as a research participant, please contact the IRB (OPRS) at. Thank you for your participation! Names Affiliation / phone number 5 Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that accommodates all possible contingencies of response One that uses specific, simple, clear language One that is pretested 6 3

4 Aspects of the questionnaire may unintentionally influence responses: Question format acquiescence response bias Including an explicit don t know response Question order effects priming 7 Yes/No question examples Would you be interested in learning about courses from the past experiences of other students who have taken them? Yes No Do you feel ready to apply what you have learned today? Yes No 8 4

5 Agree-disagree question example Administrators (e.g., principal, assistant principal, dean of students ) at this school a. Collaborate with faculty and staff to make the school run effectively. Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree 9 Acquiescence Response Bias Form A (agree I s version): Individuals are more to blame than social conditions for crime and lawlessness in this country. Form A (Agree I s) Agree (I): 59.6% Form B (agree SCs version): Social conditions are more to blame than individuals for crime and lawlessness in this country. Form B (Agree SCs) Agree (SC): 56.8% Disagree (SC): (473) Disagree (I): (472) 10 5

6 Revising yes/no questions Do you feel ready to apply what you have learned today? Yes No Do you feel ready to apply what you have learned today, or do you not feel ready to do that? Feel ready Do not feel ready 11 Revising yes/no questions Would you be interested in learning about courses from the past experiences of other students who have taken them? Yes No How interested are you in learning about courses from the past experiences of other students who have taken them? Not at all interested, Slightly interested, Moderately interested, Very interested, Extremely interested 12 6

7 Revising agree-disagree questions Administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively. Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree How well do administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively? Not well at all, Slightly well, Moderately well, Very well, Extremely well 13 No opinion or don t know options Debate about whether or not to include Explicitly offering overestimation of don t knows Respondents who are forced to pick answer provide one that has validity Recommendation don t offer explicitly but can be accepted if volunteered or written in 14 7

8 Question order Part-whole question sequences Marital satisfaction-general life satisfaction Priming Anchoring effects Sets of items that ask about a series of targets on the same response scale Ratings of how important a problem is 15 Question order Ask general questions before specific one If concerned about anchoring effects, rotate order of questions across respondents. 16 8

9 Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple, clear language One that accommodates all possible contingencies of response One that is pretested 17 Double-barreled questions: Closedended questions How clearly and concisely was the instructor able to answer questions in this course? Not at all, a little bit, somewhat, or a great deal? How would you evaluate participants materials (e.g., high quality; easy to follow; helpful, etc). Would you say very poor, poor, fair, good, or very good? 18 9

10 Double-barreled questions: Closedended questions How clearly was the instructor able to answer questions in this course? Not at all clearly, a little clearly, somewhat clearly, very clearly, or extremely clearly? How concisely was the instructor able to answer questions in this course? Not at all concisely, a little concisely, somewhat concisely, very concisely, or extremely concisely? 19 Double-barreled questions: open-ended questions How have advising services been helpful to you, including group or individual advising sessions? How could they be improved? 20 10

11 Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple, clear language One that accommodates all possible contingencies of response One that is pretested 21 Characteristics of a good question - example Indicate how the amount of use of the following types of classroom assessment have changed in your classroom Assessment during instruction to determine: degree of comprehension on part of an individual student

12 Characteristics of a good question - example Has your use of the following types of assessments increased, decreased or stayed the same? Assessment during instruction to find out how well a student has understood the material.. 23 Example Advising services: Which of these types of workshops has been or would be most helpful to you in planning your program? Please rank these from 1-5 with 1 being the least helpful and 5 being the most helpful. Inconsistencies between two sentences Ranking instructions - counterintuitive 24 12

13 Example Advising services: Below are five types of workshops that you may have attended or would attend to help in planning with your program. Please put a 1 next to the type of workshop that you think has been or would be most helpful to you. Put a 2 next to the type of workshop that you think has been or would be second most helpful. Use the numbers 3, 4, and 5 to indicate which would be the third, fourth, and fifth most helpful. 25 Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple, clear language One that accommodates all possible contingencies of response One that is pretested 26 13

14 Exhaustive response options: example How do you get your news about GSLIS? Web site calendar GSLIS weekly digest Facebook Twitter Campus bulletin boards Word of mouth 27 Exhaustive response options - example How do you get your news about GSLIS? Web site calendar GSLIS weekly digest Facebook Twitter Campus bulletin boards Word of mouth Other (please specify: ) 28 14

15 Covers whole range of options - example How long have you been a GLIS student? o Less than a month o Less than a year but more than a month o More than a year What if the respondent has been a student for exactly one year or exactly one month? 29 Covers whole range of options - example How long have you been a GLIS student? o A month or less o Less than a year but more than a month o A year or more 30 15

16 Overlapping response options - example How many times per month do you eat out a restaurant? o 0-2 o 2-4 o 4-6 o 6 or more 31 Overlapping response options - example How many times per month do you eat out a restaurant? o 0-2 o 3-5 o 6-8 o 9 or more 32 16

17 Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple language One that accommodates all possible contingencies of response One that produces variability in response One that minimizes social desirability One that is pretested 33 Pretesting Preferable to test the questionnaires with people like those in your main study population Test in same mode to be used for the study Consider cognitive pretesting 34 17

18 Other characteristics of a good question Specify who, what, when, where and how. Over what period of time? What s included? Provide cues 35 Other characteristics of a good question - example Does this table of contents cover recent developments? Does this table of contents cover recent research / topics in this field, or does it not? How many people live in your household? How many people live in your household? Please include yourself in the count, and also those who people who live with you but are away temporarily. Don t forget to count any babies and children. How many years have you been enrolled in a college or University? Not counting summers, for how many years have you been enrolled as a student in a college or university? Please count each year that you were enrolled even if these are not consecutive years

19 Open vs. Closed Questions Closed questions are usually better Easier for the respondent Less coding later Better to have respondent do categorizing Categories help define the question 37 Open vs. Closed Questions - example What grade level(s) have you worked with in the past 12 months? (e.g., 6th, 7th, 8th) Which grade level(s) have you worked with in the past 12 months? (please select all that apply). Grade 6 Grade 7 Grade 8 How often do you send text messages? How often do you send text messages? You can answer in any of the three different ways below, whichever one would be most accurate for you. You can tell us how many times per DAY you send text messages, or how many times per WEEK you send text messages, or how many times per MONTH you send text messages

20 Disadvantages of Closed Questions Categories may be leading to respondents May make it too easy to answer without thinking May limit spontaneity Not best when asking for frequency of sensitive behaviors there are numerous possible responses role of initial question development and pretesting 39 Response Options Response categories should be consistent with the question Categories should typically be exhaustive, including every possible answer Categories should be mutually exclusive Don t know category usually not appropriate knowledge questions 40 20

21 Response Options Match Question - example Did the facilitator have a thorough understanding of the subject matter? Very Poor, Needs Improvement, Okay, Good, Excellent How good or bad was the facilitator s understanding of the subject matter? Extremely good, Somewhat good, Neither good nor bad, Somewhat bad, Extremely bad 41 Response Options - example Please indicate below which of the following organizations you interact with on a regular basis. Please check all that apply. Organization A I regularly exchange ideas or advice with members of this organization. I regularly collaborate on projects with members of this organization. I have personal friends in this organization. I provide funding or receive funding from this organization. How often do you exchange ideas or advice with members of each of the following organizations? Organization A: Never, Rarely, Sometimes, Very Often, Extremely Often Organization B: Never, Rarely, Sometimes, Very Often, Extremely Often Do you have at least one personal friend in each of the following organizations, or do you not? Organization A: Have, Do not have Organization B: Have, Do not have 42 21

22 Response Options 5-7 response options General guidelines number of scale points (sensitivity vs. meaning) labeling of scale points 43 Response Option Labels - example How would you rate the instructor s ability to explain the material in this course? Extremely poor, Somewhat poor, Fair, Somewhat good, Extremely good (or Excellent) 44 22

23 Ordering the Questions First questions should be directly related to the topic as described in the introduction or advance/cover letter First questions should be relatively easy (e.g., not knowledge questions) and nonsensitive. Leave objectionable questions (e.g., income) for the end Put demographic questions at the end Usually ask general questions before more specific ones to avoid influence of the latter on the former 45 Physical Format Checklist Number all questions sequentially Use large, clear type; don t crowd White space: Place more blank space between questions than between subcomponents of questions List answer categories vertically instead of horizontally Avoid double/triple banking of response choices Be consistent with direction of response categories Be consistent with placement of response categories 46 23

24 Physical Format Checklist (continued) Don t split questions across pages. If necessary (e.g., question requires 1.5 pages), restate question and response categories on next page Put special instructions on questionnaire as needed, next to question Distinguish directions from questions be consistent 47 Example: 48 24

25 Key References To Get Started Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, Bradburn, N, Sudman, S. and Wansink, B. Asking Questions: The Definitive Guide to Questionnaire design for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, See also past issues of Public Opinion Quarterly 49 Other References Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. New York: Academic Press. Krosnick, J.A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., et al. (2002). The impact of "No Opinion" response options on data quality: Nonattitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, Saris, W. E., Krosnick, J. A., & Schaeffer, E. M. (2005). Comparing questions with agree/disagree response options to questions with construct-specific response options.unpublished manuscript, Political, Social, Cultural Sciences, University of Amsterdam