Broad selections challenge designers of multiple-choice questions

When you write non-numeric selections of a multiple-choice question, you want them to be different from each other and cover the likely choices as completely as possible. Yet at the same time you don’t want too many selections. You also would like them to be close to the same length. We have compiled other good practices.

The selections also need to be quickly and decisively understood by respondents.  Respondents don’t want to puzzle over meanings and coverage of terms. Partly that means you need to cure ambiguities but partly it means to choose terms in selections carefully so that nearly everyone interprets them the same way at first reading.

We found an instructive example in one of the law-firm research surveys. Did the questions in the plot below achieve quick clarity?  We do not know if the left-hand-side labels mirror the selections on the questionnaire. Some surveys have more detail, and even explanations, but the report gives an abbreviation of the selection.

I wonder whether most of the general counsel understand “Partner ecosystem”, let alone in the same way. Should there be a two notions joined as in “New sources of revenue and new business models”? Some companies might pursue revenue or a new business model, but not both. Likewise, why pair “Clean energy and environmental regulation”? They could be seen as two separate trends. The selection “Geopolitical shifts” feels so broad that it invites all kinds of interpretations by respondents.

This question challenged the survey designors with an impossible task. First they had to pick the important trends — and what happened to “Demographic changes”, “Big data”, “Urbanization” and “Taxes” to pick a few others that could have been included? Second, they had to describe those multifaceted, complex trends in only a few words. Third, those few words needed to fix a clear picture in lots of minds, or else the resulting data represents a blurred and varied clump of subjective impressions.

Leave a Reply

Your email address will not be published. Required fields are marked *