When you create a multiple-choice question, the order of the selections to pick from can make a difference in the results you obtain. For example, if the first selection is the most likely choice, you may suggest to some respondents that the order of the selections reflects a pattern of declining priority. Which ones they pick (and depending on the restrictions, how many they pick) will be influenced by that perceived priority suggestion.
Many surveyors alphabetize the selections as a method to counteract any such bias in their order. Another technique, which we will discuss later, randomizes the order of the selections as presented to each person. High-end survey sites can make the ordering vary for each respondent according to a randomizer. An intermediate solution would be for the surveyor to create a few different versions of the survey that each vary the orderings of the selections.
To test one aspect of whether order of selections influences respondents — the number of times a selection is picked, I analyzed a recent survey. The data consists of four multiple-choice questions that allowed respondents to check all of the selections that applied to them. Think of a whimsical question like “Which desserts do you like best? (check all that apply)” where there are seven different scrumptious delights and some or all of them could be checked.
The plot below shows totals for how many times respondents chose the first selection of the question (the bar with 1 at its base on the x axis and the height of 483), the second selection (bar 2 at height 465), and so on.
It presents the result from only their first seven selections of each question, since that was the minimum number of selections across the four questions 1. As far as I could tell, there was no evident logic to the arrangement of the selections. Each one seems plausible and not ranked in any apparent way.
A different form of this empirical inquiry would take multiple-choice questions where only one selection was permitted. Obviously, also, the selections cannot be some fact that is fixed, such as the position or age of the respondent; the selections need to invite independent, subjective judgments, as with dessert preferences.
Returning to the plot, with the exception of the fifth-position selection, the number of selections drops off steadily as the position of the selection increases. That is to say, for the most part, respondents checked selections fewer and fewer times as they moved down the seven selections. It’s as if respondents grew fatigued and didn’t pay as much attention to the later selections.
- This decision could throw off the results in that a couple of the questions had 9 or 10 selections. ↩