Moral DilemmaUsing surveys, whether on the general public or a targeted population of interest, can be an effective and fruitful way to gather information. However, there may be, at times, reasons to question whether your survey participants are being honest with their results. Concerns may arise if your population of interest is potentially over-surveyed, when you are asking about sensitive topics, or perhaps a monetary incentive to participate leads some people to speed-through the questions without offering sincere responses. Some researchers include questions specifically designed to highlight dishonesty or inconsistencies. These “lie” or “inattentive” strategies Include:
  • Attempt to determine if someone is hurrying through a survey with a question like: “Respond with ‘strongly agree’ to this question” on the scale from “strongly agree” to “strongly disagree”. If respondents choose the wrong answer, the researcher can assume that they did not read the survey questions fully.
  • Include an open-ended question or two and eliminate answers from respondents who offer especially pithy responses, such as those who write only “it’s good” or “just because”, for example.
  • In a series of multiple-choice questions, leave some check boxes without written answers next to them; if participants select blank answer slots, it could highlight the fact that they were not paying particular attention to their responses.
  • Checking for consistency by including two very similar questions, with one framed positively and one framed negatively. For example: “I usually try to sleep at least eight hours” and “I rarely get less than eight hours of sleep.”
  • Include questions with only one right answer or answers that are known to be false. For example, a survey about teen drug use included “pinwheels” (a nonexistent drug) as an option for the “select which substances you have used in the past month” question; those who selected “pinwheels” were removed from the survey; another asked about all the languages learned in a program when the program had nothing to do with learning languages.
  • Ask the survey participants at the conclusion of the survey how honestly they were able to answer the questions, on a scale from “completely honest” to “not at all honest”. This is sometimes seen on surveys about especially sensitive topics and allows the participants to take a moment to reflect on their own survey responses.
Potential Pitfalls of Using these Strategies None of these strategies come without potential costs and extra considerations: is it appropriate to include questions that will not result in useful data and may waste the time of the survey participants? Will people taking the survey and paying attention see through your false questions, and feel they are wasting their time by taking your survey? Will you need to make judgment calls about people’s honesty or effort? A related concern is that some of these strategies will reflect poorly on the survey designer. Leaving blank response options can make it look like the survey was designed in a careless fashion. Similarly, asking the same question but with opposite wording (as in “I enjoyed X” and “I did not enjoy X”) may appear equally as careless to respondents or may be confusing. Finally, it is important to consider the effectiveness of each of these strategies. A particular example of as strategy that risks being over-trusted is asking the participants to report their own honesty; if a person took a survey and felt compelled to be dishonest about their answers, what is the likelihood that he or she would admit to this in the end? Ethical Considerations By throwing out any “incorrect” responses on a lie question, the population is fundamentally changed; this makes it very important to report any adjustment strategies with the methodology of the research. It is especially important to consider what steps should be taken if a large percentage of the responses come back as inattentive or inconsistent. While this could signal a flaw in the lie question or an inattentive survey population, this circumstance certainly requires special consideration, including whether it is ethical to use the data from this population at all. Have you used any of these strategies listed here or other similar strategies in a survey before? What were the results? Continue the conversation in the comment thread! For more information on getting the best results out of your surveys, see a previous blog by project director Liz Radel Freeman on Taking Your Survey Data to the Next Level