Research surveys by law firms ask multiple-choice questions much more frequently than they ask any other style of question. They do so because it is easier to analyze the data from answers selected from a list or from a drop-down menu. Not only are they common, multiple-choice questions often permit respondents to mark more than one selection. These multi-questions, as we will refer to them, have instructions such as “Choose all that apply” or “Pick the top three.” The image below, from page 11 of a 2015 survey report by King & Wood Mallesons, states in the footnote that “Survey participants were able to select multiple options.” Thus, participants could have chosen a single selection or up to 10 selections.
To get a sense of how many multi-questions show up, we picked four survey reports we recently found and counted how many multi-questions they asked, based on the plots their reports presented. The surveys are Kilpatrick Townsend CyberSec 2016, King Wood AustraliaDirs 2015, Littler Mendelson Employer 2018, and Morrison Foerster ConsumerProd 2018. In that order they have 7 multi-questions in 24 non-Appendix pages, 4 in 36 pages, 8 in 28 pages and 4 in 16 pages. Accordingly, results from at least 21 multi-questions appeared in 104 pages. Bear in mind that each report has a cover and a back page that have no plots and almost always other pages without plots so the total number of survey questions asked is always less than the number of report pages.
While multi-questions certainly allow more nuanced answers than “Pick the most important…” questions, for example, and create much more data, those more complicated pools of data challenge the survey analyst regarding how best to interpret and present it.
A number of analytic approaches enable an analyst to describe the results, to glean from the selection patterns deeper insights, and to depict them graphically. We will explore those techniques.