Four techniques to make selections more clear

When someone creates a multiple-choice question, they should give thought to where and how to explain the question’s selections. People spend time wordsmithing the question, which is valuable time, but not the end of the matter. Even the invitation to survey participants may explain some background and key terms that shed light on selections. But at least four other options present themselves in the service of selections that can be answered without interpretative complexity.

First, a firm’s survey software should allow the designer to place an explanatory section before a question or series of related questions. That section can elaborate on what follows and guide readers in choosing among the selections. This technique has been overlooked in many of the questionnaires done for law firm research surveys.

Second, the question itself can be written carefully so that participants more easily understand the selections that follow. [This is not referring to directions such as “check all that apply” or “pick the top 3.” The point here pertains to interpretation and meaning of the multiple choices.] For example, the question might make clear the period for which answers should be given covers the previous five years. Or the question might define “international arbitration” in a certain way to distinguish it from “domestic arbitration.” The overarching definitions and parameters laid out in the question shape inform each of the selections that follow.

Third, as a supplement to the main question, some survey software enables the designer to add instructions. Using NoviSurvey, for instance, the instructions appear below the question in a box, and offer additional explanatory text. Instructions commonly urge participants not to put in dollar signs or text in a numeric field or to enter dates in a specific format, but they can also explain the selections. For example, the instructions might note that the first four selections pertain to one general topic and the next four selections pertain to a second topic. Or the instructions might differentiate between two of the selections that would otherwise perhaps be confused or misconstrued.

Finally, even if there is no explanatory section, guidelines from the question itself, or illumination in instructions, the selections themselves can embed explanatory text. Any time a selection has an “i.e.,” or an “e.g.,” the person picking from the selections should be able to understand them better. Sometimes a question will say “… (excluding a selection shown above)” to delineate two choices.

As a by-product, the more you expand on the selection choices, the more you can abbreviate them. The interplay between these four techniques to disambiguate selections, to present them more directly and clearly, allows careful designers of questions to craft selections more precisely and usefully.

Multiple observations from one multiple-choice plot

This complex, swoosh-crowned graphic spawned many observations. It graces page 34 of Pinsent Masons’ “Pre-empting and Resolving Technology, Media and Telecoms Disputes” (2016). The first several comments address the question asked, the others focus on the visualization of the findings.

  1. Time bounded. It is a good practice to limit answers to a specified period. This question asked respondents to look back five years, rather than a sloppier version like “Have you ever used any of the following institutions?” Even clearer would have been a phrasing like “in your organization during 2012 though 2016 …”
  2. Large number of selections. We can’t know how all these 28 selections were arrayed for respondents to review or whether they were in a drop-down menu. Certainly this multiple-choice question touches the outer bounds of fecundity.
  3. Choose all that apply. It would be better if the question (or plot or adjacent text) made clear that respondents could tick several institutions. Assuming they could, it is harder to interpret the percentages.
  4. Sizable ‘Other’. Even with 27 specific selections, quite a few others must not have appeared on the list (or respondents did not spot or recognize the name of one that was on the list). ‘Other’ garnered 17%, which is a larger percentage than 22 of the specific selections. It appears also that the questionnaire did not have a space for respondents to write institutions that they used but did not see among the selections.
  5. Always want more. Law firms work hard to do the best they can, yet they are usually confronted by readers who wish they had done more. Regarding the topic of this plot, for example, a question might have sought a break down between disputes before these institutions commenced by the respondent organization or brought against them by some challenger organization. We also are not told how many TMT disputes (technology, media, and telecoms) the respondents’ organizations engaged in per institution. Or the firm might have woven in data from the institutions about how many disputes they handled during the five years. Finally, the interpretation and elaboration on this set of findings is minimal.

How Pinsent Masons chose to present their findings also leads to several observations.

  1. States the question. Highlighted in a brick-red hue at the top, the question as asked on the online questionnaire lets reader efficiently match it to the answers. Interpretation of the results becomes much easier.
  2. Alphabetical order. Normally, you expect selections to be ordered in decreasing number or percentage. Here, however, with 28 different selections, putting them in alphabetical order enables readers to find institutions more easily. Note also that ‘Other’ appears at the bottom of the stack, not in alphabetical order.
  3. Monochromatic. Many designers would have splashed rainbow hues on the bars. Color just for the sake of color would impose more visual burden but add no information. On the other hand, two colors might have distinguished “institutions” from “types of arbitration.” Alternatively, since many of the selections are based on a country or regional institution, a modest color scheme by continent might inform readers.
  4. Missing information. The report incorporates submissions from an impressive 343 participants. Question number 27 was midway through an imposing set of 55 questions in the online survey. That said, we do not know how many of them tackled this particular question nor how many total ticks they made. Not knowing those particulars, when the top bar says that 15% checked that institution, we do not know the absolute number of checks.
  5. Dramatic design. Almost all the pages of the report have some variation of the red, curvy leitmotif. We admire this eye-catcher as it breaks up the white expanse, paints a touch of color on the page, and cuddles the plot itself. An attractive, simple element with aesthetic appeal.
  6. Flipped coordinates. because the names of the institutions are so long, this plot would not work well if those names had been crowded on the horizontal axis (or rotated extremely). Quite properly the plot designer rotated the plot (sometimes called “flipping the coordinates”). We also commend the firm for not duplicating the percentages along the bottom axis or cluttering the panel with grid lines.

‘Other’ selection often picked more than specific selections

In multiple-choice questions, ‘Other’ should ideally be last-resort selections. Knowledgeable questions should cover all plausible answers, which should leave little need for respondents to check ‘Other’. But that is often not true in research surveys by law firms. To the contrary, ‘Other’ quite often ends up chosen more than one of the preceding, specific selections.

Here is an example of plotting the data from a multiple-choice question that included an ‘Other (please specify)’. Unusually, it lists ‘Other’ in its ordered ranking by percentage rather than at the bottom, the conventional treatment. Clearly, five specific selections were chosen less frequently than ‘Other’, which suggests that an analysis of what respondents filled in — assuming the questionnaire offered a free-text box — might have carved some of them out and named them.

To dig deeper into this inquiry, regarding the ratio between the number of respondents checking ‘Other’ and checking the remaining selections, we analyzed four research surveys. The four surveys were Allen & Overy, “Unbundling a market: The appetite for new legal services models” (2014); Berwin Leighton Paisner, “Legal Risk Benchmarking Survey: Results and analysis” (2014); DLA Piper, “DLA Piper’s 2017 Compliance & Risk Report: Compliance Grows Up Increasing Budgets and Board Access — Point toward Greater Prominence, Independence” (2017); and Fulbright & Jaworski, “Fulbright’s Sixth Annual Litigation Trends Survey Report” (2009).

From this small and perhaps unrepresentative sample, we found that in four questions ‘Other’ received fewer checks than any of the specific selections. However, in ten questions ‘Other’ was checked more than the least-selected choice (what we termed the “smallest selection”). 1

The next plot reflects the multiple-choice questions in the four surveys that had an ‘Other’ selection. For each of those questions, the bottom axis tracks the percentage of respondents who selected ‘Other.’ The vertical axis tracks the percentage of respondents who marked the smallest of the remaining selections. The diagonal line indicates the balance point where hypothetically a question’s ‘Other’ had the same percentage as the smallest selection. Accordingly, the red dots indicate questions where more people chose ‘Other’ than chose at least one alternative selection (a few times there were two or more alternatives selected less often).

As we suggested at the start, high ‘Other’ percentages suggest that the specific alternatives could and should have been expanded. Alternatively, after the questionnaire submissions have all been collected, the firm could have tried to tease out and code the ‘Other’ mentions so that one or two of them could have been named and given specific percentages.

Notes:

  1. We noted that one firm sprinkled ‘Other’ liberally among sets of selections, yet for several multiple-choice questions with complex selections the firm chose not to have ‘Other’.  This variability seems strange.

Advantages from using multiple-choice questions

Below you can see two typical multiple-choice questions. The question comes first and then some number of selections that respondents can choose by clicking on a glyph next to the selection, such as a circle, box or underlined space.

Much of the following is based on material from the SurveyMonkey website.  Multiple-choice questions are “versatile, intuitive, and they yield clean data that’s easy for you to analyze. Since they provide a fixed list of answer options, they give you structured survey responses and make it easier for your respondents to complete the survey.” The site adds that multiple choice questions are the most common question type used on SurveyMonkey.

Multiple choice questions are fundamental to survey writing. They have many advantages for those who take your survey.

  1. Having started with multiple choice questions in grade school, respondents feel comfortable with them and may consider that format to be the “standard.” Unlike matrix (table) or open-ended questions, you rarely need to give any instructions.
  2.  Respondents can answer them with a simple click. They don’t have to formulate and write anything. And the easier you make your survey the more completed responses you’ll get.
  3. Respondents can see, review and pick from all the options with concrete definitions. For most people, having the response options in front of them is a big help. Otherwise they might overlook a plausible answer or not think of a choice.
  4. Respondents are guided as to the specificity and context for how they should answer. \footnote{I added advance good practice to drill down on a selection} For example, does someone need to report corporate revenue exactly (\$1,456,999,000) or just from a range (\$1-\$2 billion)?

From the standpoint of the law firm that conducts the survy, multiple-choice questions bring with them many advantages.

  1. Firms can shape the results of the survey. We mean this in a positive way, not a manipulative way. A law firm can focus responses in terms of answers that advance the firm’s research.
  2. Response options can subtly nudge respondents to provide more details than they would on their own. Think of a Likert scale. The more response options you provide, the more finely they quantify how much respondents agree or disagree (i.e., ‘Agree,’ ‘Agree strongly,’ or ‘Agree Somewhat’).
  3. The discipline of crafting MECE selections pushes a firm to think carefully about their topic, research goals, and plausible answers. Every firm that gets back survey responses and analyzes them comes to wish that they had refined some questions and selections more carefully.
  4. Firms can look smart and knowledgeable when the selections anticipate most respondent’s answers. You convey to participants that you know what you are talking about when you cover the likely answers. Credibility builds as the participant moves through the survey, and that credibility may sustain them to complete the questionnaire.
  5. Respondents give the style of answer you seek. Often you want to ask respondents to choose explicitly from two or more options: Do you agree or disagree? Yes or no? Should we do more, less, or the same amount? If you want a certain style of answer, provide the precise choices for your respondents to choose from.
  6. Cleaning and coding answers to open-ended questions can take a lot of work, but analyzing data from closed-ended questions is easy. The uniformity of selections allows software to aggregate responses. With open-ended responses, if someone write “alternative fees” while another writes “AFAs” and a third puts “non-hourly fee arrangements,” typical software cannot treat the three answers as the same. When you have a selection “alternative fee arrangements other than discounts,” everyone who chooses it is consistent.
  7. They look better on mobile devices. Mobile optimization is an important consideration in the survey world today. “Roughly 3 in 10 people taking SurveyMonkey surveys in the U.S. do so on a smartphone or tablet.” With such small screens and no mouse or keyboard to use, mobile devices aren’t good for surveys that use text boxes or require a lot of scrolling. .

Selections in multiple-choices ought to fit together reasonably

When a question solicits qualitative answers with selections that try to summarize concepts or ideas, the challenges are many. Among them, the question designer needs to try to pick selections that have some kind of balanced appropriateness between them. Each deserves to be reasonably similar in importance and scope, described in somewhat similar terms, and reasonably likely to be selected when read by an outsider who doesn’t know the area.

Yes, there can be technical jargon in the selections, but the outsider criterion goes to whether the selections seem facially comparable. Or at least that is my sense of how one might judge the symmetry and conceptual equivalence, so to speak, of selections.

An example should clarify this hard-two-describe quality of a good set of qualitative selections. The plot below asks a complex question of the respondents. Moreover, it seems that the three selections convey ideas of different weight and thrust and applicability.

The long and psychological “Investors are still nervous and skeptical the rally will hold” does not feel in correspondence with the stark, abstract “Neutral.” And neither of them seem to mesh with “It will be earlier than expected.” If another selection were “It will be later than expected” then readers would sense a relationship between the earlier and later pair. If “Neutral” were “No change in timing” it would more closely correspond on the scale of timing. If the “nervous” selection had a timing element, such as “Investors are nervous and will hold off” that feels more in alignment with the other selections.

It may be that my comparison of these selections leaves some people mystified or sensing subjectivity, but these multiple-choice selections just don’t feel on a par with each other. When selections do not jibe, they demand more of the respondent to process, they leave more chance of omission or overlap, and the analysis becomes more complicated.

Challenge of clear selections if they cover complicated ideas

When you write non-numeric selections of a multiple-choice question, you want them to be different from each other and cover the likely choices as completely as possible. Yet at the same time you don’t want too many selections. You also would like them to be close to the same length. We have compiled other good practices.

The selections also need to be quickly and decisively understood by respondents. Respondents don’t want to puzzle over meanings and coverage of terms. Partly that means you need to cure ambiguities but partly it means to choose terms in selections carefully so that nearly everyone interprets them the same way at first reading.

We found an instructive example in one of the law-firm research surveys. Did the questions in the plot below achieve quick clarity? 1

I wonder whether most of the general counsel understand “Partner ecosystem”, let alone in the same way. Should there be a two notions joined as in “New sources of revenue and new business models”? Some companies might pursue revenue or a new business model, but not both. Likewise, why pair “Clean energy and environmental regulation”? They could be seen as two separate trends. The selection “Geopolitical shifts” feels so broad that it invites all kinds of interpretations by respondents.

This question challenged the survey designers with an impossible task. First they had to pick the important trends — and what happened to “Demographic changes”, “Big data”, “Urbanization” and “Taxes” to pick a few others that could have been included? Second, they had to describe those multifaceted, complex trends in only a few words. Third, those few words needed to fix a clear picture in lots of minds, or else the resulting data represents a blurred and varied clump of subjective impressions.

Notes:

  1. We do not know if the left-hand-side labels mirror the selections on the questionnaire. Some surveys have more detail, and even explanations, but the report gives an abbreviation of the selection.

How common are multiple-choice questions in law-firm research surveys?

Are multiple-choice questions the most common format in law-firm research surveys? Absolutely yes would be the answer based on impressionistic leaf-throughs of some surveys. But this being a data analytics blog, we need genuine, hard-core empirical data.

Three recent surveys, picked somewhat randomly and each from a different law firm, provide data and start to suggest an answer. 1

In its 32-page report on debt financing, DLA shows 17 plots. Of them, 15 are multiple-choice with an average of 4.8 selections per question. At least five of the selections are ordinal, by the way, meaning that they have a natural order of progression. Here is an example of ordinal selections, from “Strongly agree” to “Strongly disagree”.

The Hogan Lovells report regarding Brexit has five plots throughout its 16 pages and three of them appear to be based on multiple-choice questions. The average number of selections is 3.3. Finally the K&L Gates survey of general counsel has 15 plots in its 20 pages. A dozen of them summarize multiple-choice questions with an average of more than six selections per question. 2

Combining the findings from this small sample, it turns out that 80% of the plots draw their data from multiple choice questions. The other plot types include some matrices (tables), a map, and some circle visualizations (data in circles). As to the number of selections, between four and five per question seems to be the average.

Notes:

  1. DLA Piper, “European Acquisition Finance Debt Report 2015” (2015); Hogan Lovells, “Brexometer” (2017); and K&L Gates, and “General Counsel in the Age of Disruption” (2017)
  2. We cannot determine precisely how many selections are in some of the questions because the report only shows the top three or the top five selections that were picked.

Some good practices to design multiple-choice questions

Multiple choice questions usually follow a conventional format. There is the question (sometimes called the stem) followed below by a few response selections. In line with that template, here are some recommendations for law firms or their advisors who design such questions. The graphics come from DLA Piper’s European Acquisition Finance Debt Report 2015.

  1. State the question in the form of an inquiry, not a fill-in-the-blank sentence. For example, “What is the primary method your firm uses to reduce unwanted departures of associates?” is a question. Compare it to a statement like “Our primary method to reduce unwanted departures of associates is …”
  2. Try to keep most or all of your questions in the same style, typically a question followed by selections that are each five-to-seven words (quick for readers to absorb) and something like four-to-six selections listed vertically.
  3. Have each question address a single topic. You don’t want your analysis to have to guess at the interaction of several ideas on the resulting data.
  4. Limit the number of selections. Pedagogical researchers say that three to four is the ideal number, but they are also focused on questions with a single correct answer and a handful of plausible incorrect answers.
  5.  Try to keep each of the selections approximately the same length. Don’t mix terse, one-to-three-word choices with sentence-length choices because the variability might influence which ones are picked. Here is an example from the DLA Report where the Likert scale has consistent lengths and a clear question. A Likert scale has a logical sequence of selections, unlike a question with selections that are unrelated to each other.
  6. Choose selections that are reasonable and realistic.
  7. Include clear instructions, especially if respondents are allowed to “select all that apply” or “select no more than three.” You might need to repeat the instructions before each set of questions.
  8. Avoid complicated vocabulary and technical jargon. You don’t want to cause would-be participants to drop out because the question demands too much parsing and interpretation. In further pursuit of clarity, avoid negative constructions. Here is an example of a question that might force respondents to read and re-read it.
  9. Consider whether multiple choice is the best approach. There are times when qualitative comments are ideal. Likewise, true-false and fill-in-the-blank questions may be more suitable.

Challenges with numeric selections for multiple-choice questions

Multiple-choice questions that ask survey respondents about non-numeric demographic attributes, such as position or location, leave little room for the law firm to choose a list of plausible selections. The respondent works for a company that is headquartered in a given country or continent, or the respondent has one of a relatively small number of position titles. And no one needs to pull their hair out to devise selections that will fit nearly all the respondents. An “Other” option satisfies the small clump of unusual answers.

By contrast, if the question collects numbers, such as the revenue of the respondent’s company or the number of its employees, the surveyor who decides to use a multiple-choice question must decide on the number and ranges of the selections.

We can learn more specifically from the Baker McKenzie 2016 Cloud Survey where the firm asked respondents about their company’s number of employees. I presume the selections to choose from were those shown in the donut plot below snipped from page 7 of the report.

It’s important to know that Baker McKenzie has conducted this survey in each of the two previous years, so they have some sense of the likely distribution of respondents by number of employees. Thus, they might have tailored their selections this year so that the segments of the donut plot (the percentages of companies having each range of employee) are approximately equal. That would appear to be the case from the plot.

If they had been starting from scratch, and had no knowledge about employee numbers to inform their creation of ranges, the decision would have been much more difficult. If you don’t choose ranges that match a reasonable number of your respondents, you find yourself with too many in one range or too few in another. As we have said, you can avoid this difficulty by asking the respondents to provide the actual number or an estimate (of employees or revenue or whatever). Then you create the categories based on the actual distribution of data you receive.

Categories that arise this way from the empirical data may not be as neat as categories survey reports favor. A neat category is “5,000 to 10,000 employees”. But those are arbitrary breakpoints which appeal to readers because of their symmetry, roundness, and memorability; nevertheless, when you have your data in hand you may see that different breakpoints more usefully represent the real distribution.

Reporting actual data is better than categories

Let’s give some thought to ranges checked when the surveyor wants to figure out averages or medians. The double-plot image below brings up the point. It comes from Davies Ward Phillips & Vineberg, “2010 In-House Counsel Barometer” at 9 (2010).

Considering the plot on the left, a reader should assume that the survey question was as stated at the bottom of the graphic: “Question: How long have you been a lawyer?” and that four selections were available for respondents to check. They were, one assumes, “<5 years”, “5-9 years”, “10-19 years” and “20+ years”.  Hence the pie chart plot has slices for each of those four bins.

If those assumptions are right, however, the firm could not have stated above the plot that “On average, in-house counsel have practiced law for 16.3 years …”. When a survey collects information in ranges, no analyst can calculate averages from that form of aggregated data. If 17% of the respondents have practiced law between five and nine years, it is not possible to calculate an average even for that single range let alone all four categories. So Davies Ward must have asked for a numeric answer on the questionnaire and created the four bins afterwards.

Why didn’t the firm share the more detailed information? After all, when analysts bin data, they jettison information. Furthermore, subjectivity enters in when someone allocates data to categories on the questionnaire or after the fact.

It would have been better to create a scatter plot and thereby show all the data points. That way, readers can draw their own conclusions about the pattern of the distribution.

Sometimes surveyors have concerns that individual points on a scatter plot could be tied to a specific respondent (like the longest-practicing lawyer or the highest paid lawyer). But analysts can sidestep such concerns with a box plot that tells more than the percentages in bins.