Multiple-choice questions need “Other” and MECE

On research surveys 1, many questions give respondents several choices for answers. Cleverly called “multiple-choice questions”, they probably are the most common format for questions. Given their popularity, we will devote several posts to exploring how best to create them.

Let’s pick the Hogan Lovells report released in 2014 on cross-border disputes to begin the broader discussion. On page 16 readers see the graphic shown below. We suspect the graphic presents data from a multiple-choice question in part because another Hogan survey states: “… the majority of answers [were] channelled through multiple-choice formats.”

The cross-border report does not tell the reader the question the firm asked on the survey instrument, but it may have been something like “Which of the following are [or have been] concerns of your Board in relation to cross-border disputes? (Check all that apply)”. The survey question listed the eight concerns shown in the graph, each with a checkbox or some way to select that concern. Note that the report does not explain whether the question asked respondents to check, for example, their Board’s two greatest concerns or three greatest concerns. It is most common for research surveys to invite respondents to choose as many of the selections as they want.

We commend the firm for selecting a group of very plausible Board-level concerns. However, the choices raise two observations. First, it is a good practice with a multiple-choice question to include as the final choice, “Other.” Along with that “Other” choice the question should have a text box to let respondents write in what they think was not adequately covered by the given choices. One benefit of free-form text answers is that if you do the survey a second time, your choices may be more comprehensive or more appropriately worded. A downside of text answers, however, is that they require a thoughtful human to code them, a step that injects subjectivity.

The second observation brings to mind the acronym MECE (Mutually Exclusive and Comprehensively Exhaustive). A MECE question tries to give participants choices that cover all of the possibilities (Comprehensively Exhaustive) at the same time the choices do not overlap (Mutually Exclusive). Covering all possibile choices is quite difficult (in part because you do not want a very, very long list of choices) but one advantage of having an “Other” choice is that if participants rarely choose it, you have probably done a good job of covering the waterfront. On this particular example it feels like “ability to predict outcome” and “uncertainty” overlap in some way having to do with the degree of unknown. Or, the third most common choice, “exposure/liability” blurs with the first two and the fourth choices. All four of them have to do with loss.

Notes:

  1. This post builds on my introduction to law-firm research surveys published on Nov. 29, 2017.

Leave a Reply

Your email address will not be published. Required fields are marked *