Designing a survey. Introduction. Step 1: define the aim. Step 2: Get the topics in order. Aims matter to participants too

Transcription

1 Introduction Designing a survey The goal of survey design is very simple: to get 100% of the information you need from 100% of the participants. Like all simple aims, it s kinda hard to achieve. The secret, however, is the same as any other aim: to break down the aim into achievable objectives. This exercise will take you through the process of designing a survey. Step 1: define the aim A survey is part of a research study, so the survey must match the aim of the study. Specifically, the survey must collect the data you need to answer the research question from the appropriate people. Without a clear definition of the aim, it is not clear what data you should collect. The aim should be stated as a question, and should be less than 16 words If it s longer, it s vague or complicated, or several aims. Aims matter to participants too A survey must have a clear aim that is evident to the participant. Being clear as to what the survey is about will help to maintain data quality and motivation. Having a clear aim will focus the participant s mind on the right topic and ensure the right cognitive set is in operation for the whole survey. Step 2: Get the topics in order The questionnaire is going to have to flow easily from one topic to another. The first question is what topic to start with. This is critical, because your participants will have their first experience of the survey from the first questions. So look at them carefully: - do they clearly establish the purpose of the survey? - do they make it clear that the participant s views/experiences/answers are important to the survey? - do they establish the credibility of the researchers? Here are some pointers: Beware of sensitive information 1. Never start with something sensitive. Before you ask sensitive questions you have to establish the credibility of the survey. It must be clear that you are asking for sensitive information because it is important in understanding the person s experience or point of view. Build up to sensitive areas by asking questions in which the purpose and of the survey becomes clear, and it becomes clear that the participant is taking part in an important, professional exercise.

2 Carefully introduce questions that ask the participant to disclose negative aspects of themselves 2. Never start with questions where the participant might have to disclose something negative about themselves. Again, the participant has to trust you, and believe that disclosing the information is important. You must win their confidence first. You can often to this by asking general questions before specific ones. Ask questions that show that you can understand why people might do things that they are reluctant to admit. If you are surveying hand hygiene in a medical setting, start with something like Health workers often find themselves carrying out hand hygiene less often than they should. In your own experience, what is the most important reason for this? Followed by a list of reasons. While it s interesting to know the person s reasons, it s also valuable because it is suggesting that you understand the problem and you are taking their advice on the most important issues in tackling it. When you get around to asking them about their own hand hygiene, they will know that you are being non-judgemental, and that you are asking for their help in tackling the problem. The idea here is to make the person feel that their answers will help to make a difference, that their experience is important in understanding the situation, and, above all, that they are dealing with a person who understand why they might do things that they should not, or fail to do things that they should. The five-to-one rule Every time you ask something that is intrusive, or makes the person admit to something they would rather not admit to, you are creating a negative experience. Studies of married couples have shown that couples who stay married are not ones who never say anything negative, but they are couples who say at least five positive things for every one negative thing. The same applies in surveys. You need five feel-good questions where the person feels that their opinion is important, or they can say something good about themselves, for every one occasion on which they will have a negative experience answering the question. Make sure you remember this. Engage instantly never start with anything dull Here is an error that occurs again and again. Researchers start writing their results section by describing the participants age, sex, social class, income etc. The trouble is that these questions are dull, intrusive (why does this person want to know my income??) The person must engage with the purpose of the survey instantly. You will start the survey by explaining its purpose. So if your survey is about parents beliefs and practices around feeding school-age children, then start with the topic! Ask general questions, for example

3 Do they think that schoolchildren nowadays eat better or worse than when they were young? Do they think that parents have less time and opportunity now to prepare healthy food for their children? Do they feel that children are exposed to a lot of advertising for unhealthy foods? These sorts of questions do two things: 1. They frame the survey. The person understands what the survey is about, and that their opinions are important. They start the person thinking about the survey topic in a way that is non-threatening. If you want to get a person to like you, ask them their opinion on anything and listen attentively. (This works in social situations too.) 2. They suggest that there are all sorts of reasons why parents may not be able to feed their children the diet they would like to. This paves the way to asking about what the parents think their children should eat, and what their children actually eat. Leave dull topics to the end, and shorten them If you want people to answer dull questions like what age they were when they got marred, or where they were born, then make sure that every question is vitally important for your study. Leave them to the end of the questionnaire or interview. If you can, put them immediately after a feelgood section, where the person will have enjoyed answering the questions. Try introducing them with a thank-you: Thank you for taking part in this survey. To finish, just a few details about you yourself: Decide on your topic order 1. Divide the survey into topics 2. Which topic is a good starter? 3. Which topic(s) will have to wait until you have gained the interest and trust of the participant? 4. Which topics are dull? Draw up a provisional order for the topics in the survey. Do they flow naturally? Or will you need a little bit of connecting material to join them?

4 Step 3 : Item writing Identify items you can copy The first step in question writing is not to write them, but to source them from previous research. For example, many population health surveys use standard items that measure quality of life and wellbeing. These items often originate with organisations such as the WHO, or the US Behavioral Risk Factor Surveillance System (BRFSS), which is run by the US Centers for Disease Control. SLÁN-02; BRFSS A3 In general would you say your health is...? Excellent... 1 Very good... 2 Good... 3 Fair... 4 Poor... 5 BRFSS A4 Is your daily activity limited by a long term illness, health problem or disability? Yes 1 No 2 SLÁN-02; BRFSS A5 During the past 30 days, for about how many days did poor physical or mental health keep you from doing your usual activities, such as self-care (that is looking after yourself), work or recreation? Number of days None 0 The example shows the Irish National Health and Lifestyle 2006 survey, SLÁN-06. You can see that these three important questions use a format from the BRFSS surveys, which had also been used in the 2002 survey, SLÁN-02. It s important to be familiar with the literature, and where possible to use questionnaire items that allow key comparisons to be made. In the case above, it was very important to be able to compare the responses to these questions with the responses in the previous survey, but it was also important to be able to compare them with figures from other countries. Which comparisons are important to you? Try to base as much of your questionnaire as possible on items that will allow you to make the most important comparisons. In the case of SLÁN, the most important comparisons were with previous versions of the survey, while international comparisons were less important (but important nonetheless). Are comparisons more important than accuracy? Sometimes you may find yourself in the dilemma that the survey you want to compare with used a question that was poorly written and that you believe will give an inaccurate estimate of whatever you are measuring. Should you use the same question or write a better one? There is no definitive answer, but there need to be very good reasons for putting a bad question onto a survey! In one case, I was involved in a survey

5 where the previous survey had used a question that almost certainly underestimated the prevalence of smoking. We decided not to use the question from the previous survey because our priorities were 1. Most important: gather accurate information 2. Next most important: make comparisons with the previous survey 3. Least important: make international comparisons. With this order of priorities, it was clear that we should put in a question on smoking that gave the best estimate of prevalence rather than stick with the format of the previous survey. Writing your own items Aligning your questions It is important to align the purpose of the question and the format of the question. Think forward to your final report. What kind of data will you need in order to write the report? Make sure that the question captures the information that you need. It goes without saying that you must not ask questions that don t have a role in your final report. But many researchers find themselves with questions on their survey that just seem to have arrived from nowhere. I once did a paper on boredom-proneness after I discovered that a large survey of elderly people had a question on boredom. But the fascinating thing was that no-one knew why the question was on the survey, or even who had put it there! How much detail? Survey questions often have too many possible answers. We survey people with questions that use a large numbers of categories, and as soon as the data analysis is started, the first step is to combine categories! For example, the original questionnaire may use a question like this: How often in the last month have you had difficulty in falling asleep? Never # Seldom #Sometimes # Often # Very often but very often the researcher will combine the often and very often categories and simply report on the percentage of people who often had difficulty in falling asleep. If you are going to report it as a simple binary variable, then ask the question like that! In the last month have you often had difficulty in falling asleep? Yes # No

6 Studies of questionnaire responses tend to show that questions with 2 possible answers are faster, are less likely to be skipped, and give data that is just as good in quality as 5-point scales. Writing a good 2-answer question The trick in writing a good two-answer question is to make the question strong. A strong question is one that has a strong definition of the thing you are trying to measure. Here is a stronger version of the insomnia question: In the last month have you often taken more than an hour to fall asleep after you go to bed? Yes # No With a stronger definition like this, you will have a lower prevalence, but you will also have a more precise definition of insomnia. 5-point scales ever? Use 5-point scales if you really need to measure the severity, or the frequency. But if you are just measuring the presence or absence of a problem, or an attitude or a behaviour, then use a 2-answer format. Phrasing your question This may sound silly, but a question should end with a question mark. But you do see questions written as run-on items: Run-on item: Was the pack provided by # a nurse # a doctor # a health visitor # can t remember Question format Who provided the pack? # a nurse # a doctor # a health visitor # I can t remember Keep it short and simple Keep questions short. Scan them for material that can be omitted:

7 Original item Which of the following best describes your normal walking pace? # a slow pace # # a steady average pace # a fairly brisk pace # a fast pace Simpler wording What speed do you normally walk at? # slowly # average speed # briskly # fast The first thing to get rid of is of the following. It tells the person nothing that they don t know that the possible answers follow the question. Second, the word pace occurs five times in the question and the answers. The question is really about speed, which is a simpler word than pace. The whole question can be shortened by 50% from 24 words to 12, making it quicker to do and easier to read. Presentation and layout It s worthwhile spending time on presentation and layout. The visual appearance of your questionnaire will be the first factor that a participant will notice. It can motivate them to take part, or to put it in the bin. So what do they notice? Is it interesting? In a meta-analysis of factor affecting response rates, the most important predictor of whether a person will do a questionnaire or not is whether it looks interesting or not. Give the questionnaire a short, meaningful and attractive title that will be the first thing that the participant reads. And make sure that the first questions hook the participant s attention. If the survey is about sleep quality, start by asking about sleep. As I said, keep dull or intrusive questions till your questionnaire has established some rapport with the participant. Surveys containing sensitive questions are less likely to be filled in, so be careful with how you position these items. Does it look professional? Having good quality paper with a colour logo of your institution (get their permission!) will make the questionnaire look professional. There is some literature on whether printing on coloured paper improves response rates, and the answer is that it doesn t seem to matter. Questionnaires originating from universities are more likely to be filled in than ones from commercial organisations.

8 Does it look short? If your questionnaire is just a single sheet of paper, it s more likely to be answered than if it s a thick sheaf of paper. Furthermore, data quality gets worse towards the end of long surveys. Try to keep it short. Try to make sure that your survey doesn t get longer than you can print on two A4 sheets that can be folded in the centre to make a sort of 4-page booklet. Shorter questionnaire, as you might expect, has higher response rates. Can they read it? Remember that many older people have trouble reading print smaller than 12 point, even with their usual glasses. Do not use print smaller than 12 point. Make sure that response options are lined up properly with their check boxes. DO NOT WRITE IN CAPITALS Capital letters are 20% less legible than lower-case letters. They also give the impression that you are shouting. If you have something important, write it in bold, not capitals. Further reading Fanning, E. (2005). Formatting a paper-based survey questionnaire: Best practices. Practical Assessment, Research & Evaluation, 10(12), 2. Team work The aim You are planning to do a survey of people who were suffered a heart attack. You want to find out the prevalence of anxiety and depression, and to identify the factors associated with these. Participants will be given a questionnaire at routine follow-up outpatient visits. Key predictors include time since the attack further coronary admissions since the attack presence and severity of angina socioeconomic status marital status the person s perception of their health status The aim of the survey, then, is to measure depression and to record each of the risk factors.

9 The questionnaire The questionnaire has two parts: the first section measures the risk factors. The second part is the Hospital Anxiety and Depression Scale The risk factor section Your team should draft the risk factor section of the questionnaire. Place each question on a PowerPoint slide, which you can present to the class. The HADS The second part of the exercise will be translating the HADS.