Challenges with numeric selections for multiple-choice questions

Multiple-choice questions that ask survey respondents about non-numeric demographic attributes, such as position or location, leave little room for the law firm to choose a list of plausible selections. The respondent works for a company that is headquartered in a given country or continent, or the respondent has one of a relatively small number of position titles. And no one needs to pull their hair out to devise selections that will fit nearly all the respondents. An “Other” option satisfies the small clump of unusual answers.

By contrast, if the question collects numbers, such as the revenue of the respondent’s company or the number of its employees, the surveyor who decides to use a multiple-choice question must decide on the number and ranges of the selections.

We can learn more specifically from the Baker McKenzie 2016 Cloud Survey where the firm asked respondents about their company’s number of employees. I presume the selections to choose from were those shown in the donut plot below snipped from page 7 of the report.

It’s important to know that Baker McKenzie has conducted this survey in each of the two previous years, so they have some sense of the likely distribution of respondents by number of employees. Thus, they might have tailored their selections this year so that the segments of the donut plot (the percentages of companies having each range of employee) are approximately equal. That would appear to be the case from the plot.

If they had been starting from scratch, and had no knowledge about employee numbers to inform their creation of ranges, the decision would have been much more difficult. If you don’t choose ranges that match a reasonable number of your respondents, you find yourself with too many in one range or too few in another. As we have said, you can avoid this difficulty by asking the respondents to provide the actual number or an estimate (of employees or revenue or whatever). Then you create the categories based on the actual distribution of data you receive.

Categories that arise this way from the empirical data may not be as neat as categories survey reports favor. A neat category is “5,000 to 10,000 employees”. But those are arbitrary breakpoints which appeal to readers because of their symmetry, roundness, and memorability; nevertheless, when you have your data in hand you may see that different breakpoints more usefully represent the real distribution.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.