Psychology 312: Lecture 20 Slide #1 No Slide #2 Overview Advantages & disadvantages. Steps involved in survey design. Question construction. o Open-ended. o Fixed alternative. o Combination. o General recommendations. We will begin with a discussion of the advantages and disadvantages associated with this particular design approach. We will then provide an overview of the general steps involved with survey design. Finally we will spend considerable time talking about the actual question construction in a survey instrument including a discussion of open-ended questions, fixed alternative questions or the use of a combination of the two. We will conclude with some general recommendations related to the overall design of a survey instrument. Slide #3 Advantages to surveys o Study behaviors one cannot (or difficult) to directly observe. Access self-report data (feelings, perceptions, beliefs/attitudes). Private behaviors or behaviors of a sensitive nature. o Relatively easy to administer. o Economical. There are a number of advantages associated with survey research. Perhaps the most importance of these is the fact that this particular approach allows us to study behaviors that we cannot directly observe or at the very least would be very difficult to directly observe. We see this really in two different instances. The first would be a case in which as a researcher we are primarily interested in self-report data. Self-report data refers to things like participant s feelings or perception or reported beliefs or attitudes about a particular topic. The second situation would be one in which we are dealing with what we might call private behaviors or behaviors of a sensitive nature. This would include things like sexual behaviors or drug taking or perhaps criminal activities. In addition to this very significant advantage surveys are also attractive because they are typically regarded as relatively easy to administer once the instrument has been developed. Finally they are generally regarded as economical. Meaning that relative to some of
the other design arrangements we have discussed in this course surveys are often seen as relatively cost effective. Slide #4 Disadvantages o Self-report data can be unreliable. o Designing a good one is challenging. With those advantages comes some disadvantages. It may have already occurred to you that one of the challenges of survey research is being confident in the reliability of self-report data. Sometimes we question the reliability of this type of data. Either because participants are reluctant to answer questions true fully or simply because they may not be that accurate at reporting upon their own behavior. Finally a significant disadvantage associated with surveys is that designing a good instrument is incredibly challenging. We will talk about survey construction for much of this lecture a little bit later, but at this point simply recognize that designing a good instrument, an instrument that can produce both reliable and valid data often times takes considerable time, effort and many revisions. Slide #5 Simple 6 steps involved in designing survey research. We will know turn to the six generic steps involved in designing survey research. In doing so we will highlight the six decision points that a researcher must consider when he or she chooses to use a survey approach. Slide #6 Step 1: state broad objectives. o Why is this survey necessary? o EX: interested in alcohol consumption among college students. Step one of survey research includes identifying the broad objectives of the study. In this case the researcher is asking why is this particular survey necessary. Notice that this is not unlike other design arrangements we have discussed over the course of this semester. In all cases the researcher typically begins with a set of broad objectives that then become increasingly narrowed into a testable hypothesis. Let s imagine for the purposes of demonstration that we are interested in examining alcohol consumption among college students. We are interested in this topic, because we see it related to other issues that are also of interest. Including things like students academic success, their mental or physical health and perhaps their personal safety. Slide #7
Step 2: narrow to a specific focus. o State particular hypothesis to be explored. o EX: does alcohol consumption increase with the number of years spent in college? In step two we will take the broad objectives from step one and narrow them into a more specific focus. This will allow us to state the particular hypothesis that we intend to explore with our survey. In the present example let s imagine that our specific hypothesis becomes does alcohol consumption increase with the number of years spent in college? Slide #8 Step 2: narrow to specific objectives. o Hypothesis will guide the type of questions asked. Quantities of alcohol? Type of alcohol? Time and context? Body weight of participants? Gender of participants? o Hypothesis will guide the comparison made. Could compare answers to specific questions. Could compare aggregated scores across several questions. Step two is crucial, because once the hypothesis has formally been stated it will guide the types of questions we will ask on our survey instrument. Specifically we might want to ask questions about the quantities of alcohol consumed. The type of alcohol consumed the time and context of alcohol consumption. In addition perhaps some information about the body weight of our participants and perhaps some information about their gender. In addition the hypothesis will also guide the comparisons that we will plan to make when analyzing our survey data. For instance we might choose to compare answers to specific questions as a means of answering our formal hypothesis. Alternatively we might choose to compare the aggregative scores across several questions that are all intended to get at the same variable. Slide #9 Step 3: determine the population & sample. o To which group would we like to generalize? o Determine size of sample. o Determine how to select sample. Random selection? Quasi-random selection?
In step three we determined the particular population and sample of our survey study. In doing so we ask to which group we would like to be able to generalize our results at the end of the study. The answer to this question will in turn determine the appropriate size of our sample and how we plan to select that sample from the overall population. Let s imagine in our hypothetical study examining alcohol consumption among college students that we plan to have are overall sample be WSU students. From that overall population we could use random selection to select particular students as participants. Random selection would mean that every student currently enrolled at the university would have an equal probability of being selected for participation. However we might opt for a slightly different strategy called quasi-random selection. In quasirandom selection certain parameters are put in place related to the overall population and then we randomly select participants out of those restricted parameters. For example if we wanted to ensure that we have the same number of participants at the freshmen, sophomore, junior and senior level we might opt to break the overall population into those different class standings and then randomly select participants from each of those individual blocks. Slide #10 Step 4: select survey protocol. o Face to face interview? o Telephone? o Mail. o Email & the web? In step four we will select the survey protocol that we plan to use to collect our data. There are several possible option. For instance we might opt to administer our survey through a face to face interview or perhaps through a telephone interview or perhaps send the survey through mail to our potential participants or finally to email participants and direct them to a website where they can access the survey. All of these are associated with specific advantages and disadvantages. All of which are discussed in detail in your text. Slide #11 Step 5: choose types of questions & responses. o Open-ended? o Both? NOTE: this decision will affect your analysis. Relates back to scales of measurement. In step five we will choose the particular types of questions and responses that will appear on the survey instrument. Like the previous step there are several possible options here. Including open ended questions, fixed alternative questions or a combination of the two. Please keep in mind that the decisions we make about the types of questions and responses on the survey itself effect the analysis that can be conducted at the end of the study. This relates back to our discussion of scales of measurement. You will recall that different forms of data whether they be nominal,
ordinal, interval, or ratio are amendable types of analysis and we want that analysis to be able to relate back to our formal hypothesis at the end of the study. Slide #12 Step 6: select analysis plan. o Relates back to hypothesis. EX: alcohol consumption and years spent in college. Statistical analysis? Correlation. In step six we will formally select that analysis plan. As just mentioned we want the analysis plan to relate logically back to the hypothesis that drove development of the study in the first place. For example in our hypothetical study examining alcohol consumption and years spend in college the appropriate statistical analysis would probably be a correlation. Note that in this case we are now attempting to manipulate either variable. Either alcohol consumption or years spent in college. Instead we are simply trying to determine whether changes in one of these variables is associated with changes in the other variable. A correlation is the appropriate analysis to answer this question. Slide #13 Open Ended. o No fixed set of potential answers. In what situations are you most likely to consume alcohol? What events tend to cause you to consume alcohol? Now let s turn to a discussion of the types of questions that can be used in survey research. One category includes open end questions. As their name suggests open ended questions are questions that do not include a fixed set of potential answers. Two possible open end questions that could be asked in our hypothetical study might be in what situations are you most likely to consume alcohol or what events tend you to consume alcohol? Slide #14 Advantages: o Don t impose a particular point of view. o May provide info researcher had not considered. Disadvantages: o Difficult to analyze. o How to translate answers into categories that fit hypothesis? What events tend to cause you to consume alcohol? (NOTE: event be interpreted in different ways.)
Advantages associated with open ended questions include the fact that they do not impose a particular point of view on the participants. As a result they may provide information to the researcher that he or she did not consider previously. Disadvantages associated with this particular approach. Include the fact that open ended questions may be difficult to analyze. It may be challenging to determine how to translate answers into categories that relate back to the original hypothesis. For example in our hypothetical study if we were to ask participants what events tend to cause you to consume alcohol. It is possible that our participants will interrupt the term event in many different ways. Slide #15 Fixed alternative. o Participant picks from a set of possible responses. Do you drink at social gatherings with friends? Yes. No. Do you drink at social gatherings with friends? Yes, always. Yes, sometimes. Yes, but rarely. No. The second category that can be used as survey research includes fixed alternative questions. In a fixed alternative question the participant answers the question by picking from a set of possible questions Slide #16 Advantages: o Easy to translate into categories of responses. o Generally easy to analyze. Disadvantages: o Limits number of responses. o Won t know reasons for reasons. If answer yes, sometimes we won t know why. The advantages and disadvantages associated with fixed alternative questions are typically the opposite of those associated with open ended questions. For example the fixed alternative question is typically easy to translate into categories of responses. That can then be analyzed in reference to the original hypothesis. However these types of questions obviously limit the number of responses a participant can give. In addition we do not necessarily know the reasons why participants answer questions in a particular manner. For example in the previous set of questions if a participant answered yes sometimes to the question do you drink in social gatherings with friends we would not necessarily know what the mediating factor is that
determines when that individual chooses to drink alcohol and when he or she chooses not to drink alcohol. Slide #17 Combined open & closed ended. o Can pick from fixed responses or provide one not listed. Which type of beverage do you prefer? Beer? Wine? Other: Please specify. For these reasons a researcher will sometimes opt to use a combined open close ended question. This type of question takes the advantages of the open ended question and combines them with the advantages of the fixed alternative question. In this case a participant is allowed to pick from a set of fixed responses or provide his or her unique response. For example imagine that our hypothetical study we ask participants, which type of beverage do you prefer. We provide a fixed alternatives of beer or wine, but if neither of those options is really appropriate for our participant than that participant is allowed to specify his or her own unique answer. Slide #18 Advantages: o Same as open ended & fixed alternative. Disadvantages: o May still be difficult to classify open-ended answers. o But- fixed alternative options may serve as examples that guide responses on the open ended option. As mentioned previously the combined question is intended to capitalize on the advantages associated with open ended and fixed alternative questions. Unfortunately the downside of this question is that it still may be difficult to classify the open ended answers. Having said that it is also widely believed that the fixed alternative options can serve as examples that guide as responses to the open ended questions. Thus reduce the amount of variability in the types of answers participants give. Slide #19 Funneling. o Begin with an open-ended question. o Follow w/more specific response alternatives. How much time do you spend drinking? Would you say you spend most, a lot, some, a little, or none of your time drinking?
Another question strategy that combines elements of open ended and fixed alternatives questions is a strategy known as funneling. In funneling we begin with an open ended question and then follow it with more specific response alternatives. A concrete example of funneling would be the following: How much time do you spend drinking? Would you say you spend most, a lot, some, a little, or none of your time drinking? Slide #20 Advantages: o Get participant to think about answer & how this fits into specific alternative before selecting an answer. Disadvantages: o Same as fixed alternative. The major advantage associated with funneling it gets the participant to think about answer and how this fits into the specific alternative before selecting an answer. The disadvantages associated with this design strategy are similar to those associated with fixed alternative questions in general. Namely that you restrict the way in which the participant can answer the question. Slide #21 Creating Response Alternatives For fixed-alternatives questions: o Choose response alternatives that fit the nature of the question. How satisfied are you with your academic experience at WSU? Never. Rarely. Bad Sometimes. Often very often. Very satisfied. Somewhat satisfied. Not very satisfied. Not at all satisfied. Better Now let s turn to a discussion of creating response alternatives. In this case we will focus on fixed alternative questions construction. For example we want to be sure that we choose response alternatives that fit the nature of the question. Let s imagine that we ask our participants how satisfied are you with your academic experience at WSU. In this case the options on the right are more appropriate then those on the left. This is because the options on the right speak specifically to level of satisfaction. While those on the left seem to speak to the frequency of satisfaction.
Slide #22 Creating Response Alternatives For fixed-alternative questions: o Make sure answer categories do not overlap. How many alcoholic drinks do you consume in a typical week? 0-1. 1-2. Bad 2-3. 3 or more. 0-1. 2-3. More than 3. Better We also want to be sure that the answer categories in our fixed alternative question do not overlap. For example if we asked how many alcoholic drinks do you consume in a typical week. The options on the right are better than the options on the left, because those on the left overlap. In this case if we had a particular participant who say consumed on average on drink per week that participant might answer this question by choosing the first option or by choosing the second option. Slide #23 Creating Response Alternatives For fixed-alternative questions: o Can use likert-like scales (for attitudes & beliefs). Binge drinking poses significant safety risks for those who engage in it. A popular design and fixed alternative question is the use of the likert- like scale. These types of scales are often times used with questions related to attitudes or beliefs. For example we might ask our participants their level of agreement with the following statement: Binge drinking posses significant safety risks for those who engage in it. In this case the response alternatives sit on an alternatives sit on a continuum from strongly disagree to strongly agree and participants would indicate there level of agreement with the statement by choosing one of those response alternatives along that continuum. Slide #24 Creating Response Alternatives For fixed-alternative questions: o Can use semantic differential. Uses bipolar adjectives placed at each end of point scale. Rate the instructor s lecture style: Good or bad?
A slight variation on the same idea is the use of the semantic differential. In this design we use bipolar adjectives and place those adjectives at either end of a point scale. Participants then indicate there answer by selecting a particular point along that continuum. Slide #25 General Considerations: Survey Questions. Avoid ambiguity. o EX: over the last few years vs. over the last two years. Consider perspective of participant. o Interpretation? o Frame of reference? EX: what is your self-worth? In terms of financial worth? In terms of psychological worth? In addition to considerations related to the construction of fixed alternative questions. There are also some considerations that refer to survey questions in general. For example we must avoid ambiguity in our questions. The phrase over the last few years is ambiguous and could lead to multiple interpretations. It would be better to replace it with something more specific such as over the last two years. Closely related to this we must consider the perspective of our participant s. Their possible interpretations and there possible frame of reference. For example what if we ask participants what is your self-worth. They might interrupt this question in terms of financial worth or they might interpret it in terms of psychological worth. Obviously those two interpretations will lead to vastly different answers. Slide #26 General Considerations: Survey Questions. Only ask what we need to know to address our hypothesis. Especially if asking about sensitive/private info. o EX: ask about salary. Better of ask participants to choose among different ranges of salary (e.g., 35,000-45,000) rather than ask for specific number. In addition we should ask only what we need to know in order to address our hypothesis. This is particularly true if we are asking questions about sensitive or private information. For example imagine that we would like to ask a question about salary. Salary is widely regarded as highly private and many people are reluctant to openly disclose it. In this case it would be better to ask participants to choose among different ranges of salary then to ask them to report a specific number. Slide #27 General Considerations: Survey Questions. Don t make questions too long. Avoid wording that is biased or leading.
o How hostile was participant A s response to the situation? Avoid assumptions about knowledge or vocabulary. o Do you know someone whose professional career was advanced through nepotism? We want to be sure that we don t make questions too long. Long questions are often confusing for participants. We want to avoid wording that is biased or leading. Such as how hostile was participant A s response to the situation? We also want to avoid assumptions about knowledge or vocabulary. Such as do you know someone whose professional career was advanced through nepotism? Slide #28 General Considerations: Survey Questions. Avoid double negatives. o When are you least likely to feel least adequate at your job? (bad) versus o When are you most likely to feel adequate at your job? (better) We also want to be sure to avoid double negatives. For example the question when are you least likely to feel least adequate at your job?? Is necessarily confusing. It would be better to refrain that question as when are you most likely to feel adequate at your job. Slide #39 General Considerations: Survey Questions. Consider order effects for questions o EX: asking all positive questions before negative questions. Finally we want to be cognizant of any order effects that might arise from the arrangements of questions on the survey instrument. For example we would want to be sure that we did not ask all positive questions about a particular topic before asking a set of negative questions. For instance if we ask participants first to respond to a set of questions related to what they enjoyed about drinking followed by a set of questions about what they disliked about drinking Slide #30 Next Lecture That concludes this lecture. Next we will discuss single-subject designs. That concludes this lecture. Next we will discuss single-subject designs.