In multiple-choice questions, ‘Other’ should ideally be last-resort selections. Knowledgeable questions should cover all plausible answers, which should leave little need for respondents to check ‘Other’. But that is often not true in research surveys by law firms. To the contrary, ‘Other’ quite often ends up chosen more than one of the preceding, specific selections.
Here is an example of plotting the data from a multiple-choice question that included an ‘Other (please specify)’. Unusually, it lists ‘Other’ in its ordered ranking by percentage rather than at the bottom, the conventional treatment. Clearly, five specific selections were chosen less frequently than ‘Other’, which suggests that an analysis of what respondents filled in — assuming the questionnaire offered a free-text box — might have carved some of them out and named them.
To dig deeper into this inquiry, regarding the ratio between the number of respondents checking ‘Other’ and checking the remaining selections, we analyzed four research surveys. The four surveys were Allen & Overy, “Unbundling a market: The appetite for new legal services models” (2014); Berwin Leighton Paisner, “Legal Risk Benchmarking Survey: Results and analysis” (2014); DLA Piper, “DLA Piper’s 2017 Compliance & Risk Report: Compliance Grows Up Increasing Budgets and Board Access — Point toward Greater Prominence, Independence” (2017); and Fulbright & Jaworski, “Fulbright’s Sixth Annual Litigation Trends Survey Report” (2009).
From this small and perhaps unrepresentative sample, we found that in four questions ‘Other’ received fewer checks than any of the specific selections. However, in ten questions ‘Other’ was checked more than the least-selected choice (what we termed the “smallest selection”). 1
The next plot reflects the multiple-choice questions in the four surveys that had an ‘Other’ selection. For each of those questions, the bottom axis tracks the percentage of respondents who selected ‘Other.’ The vertical axis tracks the percentage of respondents who marked the smallest of the remaining selections. The diagonal line indicates the balance point where hypothetically a question’s ‘Other’ had the same percentage as the smallest selection. Accordingly, the red dots indicate questions where more people chose ‘Other’ than chose at least one alternative selection (a few times there were two or more alternatives selected less often).
As we suggested at the start, high ‘Other’ percentages suggest that the specific alternatives could and should have been expanded. Alternatively, after the questionnaire submissions have all been collected, the firm could have tried to tease out and code the ‘Other’ mentions so that one or two of them could have been named and given specific percentages.
- We noted that one firm sprinkled ‘Other’ liberally among sets of selections, yet for several multiple-choice questions with complex selections the firm chose not to have ‘Other’. This variability seems strange. ↩