Ten pitfalls of respondents on multiple-choice questions

Before plunging into the bog of blunders, let’s define respondent as someone who presses submit at the end of an online questionnaire. An alternative term would be participant. Potential respondents who stop before the end of the questionnaire are partial participants. Typically, survey software logs the responses of partial participants. Now, enter the bog, if ye dare!

We have listed below several things that can go wrong when people tackle multiple choice questions. The pictorial summarizes the points.

  1. Reverse the scale. With a question that asks for a numeric value, as in a table of actions to be evaluated on their effectiveness, a “1” checked might indicate “wholly ineffective” while a ten might indicate “highly effective.” Some people may confuse the scale of low to high and check a “1” when they mean “highly effective”.
  2. Misread the question. Hardly unique to multiple-choice questions, simple misunderstanding of the inquiry dogs all survey questions. If the question addresses “effective actions” and someone reads it as inquiring about “ineffective actions”, all is lost.
  3. Misread selections. This pitfall mirrors misreading questions, but applies to the multiple selections. Negative constructions especially bedevil people, as in “Doesn’t apply without exception.”
  4. Misread instructions. This mistake commonly appears when questions ask for a number\index{number answer}. Careful survey designers can plead with respondents to tell them “Only numerals, not percent signs or “percent”. The guidance can clearly state “do not write ranges such as “3-5” or “4 to 6”, do not add “approx..” or ” ~ .” For naught. Or people sprinkle in dollar signs or write “2 thousand” or “3K”. Humans have no trouble understanding such entries, but computers give up. If an entry is not in the right format for a number, a computer will treat the entry as a text string. Computers can’t calculate with text strings. Fortunately, computers can be instructed to scrub the answers so that they are in a standard format. And sometimes the survey software can check the format of what’s entered and flash a warning message.
  5. Fill in nonsense when answers are required. Some participants can’t be bothered to waste their time on irrelevant questions, so they slap in the first selection (or some random selection). Unless the analyst takes time to think about the likelihood of a given answer in light of other answers or facts, this mistake eludes detection.
  6. Give contradictory answers. Sometimes a survey has two questions that address a similar topic. For example, the survey might ask respondents to check the cost management techniques they have tried while a later question asks them to rate those techniques on effectiveness. What if they rate a technique they didn’t say they had tried, or they fail to rate a technique that they had tried? This could be a form of contradiction.
  7. Become lazy. When there are too many questions or selections for questions go on and on or reasonable answers require digging, respondents can throw in the towel and make sloppy selections. Here the fault lies more with the survey designer than with the survey taker.
  8. Too quickly resort to “Other”. A form of laziness, if the selections are many or complex, some people just click on “Other” rather than take the time to interpret the morass. If they write a bit about what “Other” means, that text will reduce the adverse effects of the lack of discipline.
  9. Mis-click on drop-downs. If you find a “United Emirates” in your corporate headquarters data and nearly everyone else is “United States”, you can suspect that one person made a mistake on the drop-down list.
  10. Pick too many or too few. If they pick too many selections, the software might give a warning. Otherwise, if “select no more than three” governs, the software might simply take the first three even if four or more were checked. The survey software should be able to give a warning if this mistake happens.

Leave a Reply

Your email address will not be published. Required fields are marked *