Let’s remind ourselves of what we are calling “multi-questions.” The next plot, from a research-survey report (Kilpatrick Townsend CyberSec 2016 [pg. 7]) illustrates one. The plot derives from a multiple-choice question that listed seven selections and where “More than one choice permitted” applied. The plot gives the percentage of the 605 respondents who chose each selection.
You can spot such multi-questions because the percentages in the plot add up to more than one hundred. Here they total 237% which means an average of 2.37 selections per respondent.
Now, about presenting the results of multi-questions. Other than prose, the simplest description of the distribution of responses to a multi-choice question is a table. A table succinctly tells how many respondents chose each selection. From the data set we have been using and the question’s nine selections, the total number of roles selected was 318 from 91 respondents. A maximum of 819 possible selections could have been made if each respondent had checked each selection. When you know the number of participants in your survey, you can add a column for percentages.
If a table is not sorted by a relevant column, like the table above is sorted on “Selected”, it is harder for readers to compare frequencies. Column charts use bar height to help with comparisons, as the plot below illustrates. We used the data in the table above and added the frequency of selection in each bar.
Research surveys by law firms ask multiple-choice questions much more frequently than they ask any other style of question. They do so because it is easier to analyze the data from answers selected from a list or from a drop-down menu. Not only are they common, multiple-choice questions often permit respondents to mark more than one selection. These multi-questions, as we will refer to them, have instructions such as “Choose all that apply” or “Pick the top three.” The image below, from page 11 of a 2015 survey report by King & Wood Mallesons, states in the footnote that “Survey participants were able to select multiple options.” Thus, participants could have chosen a single selection or up to 10 selections.
To get a sense of how many multi-questions show up, we picked four survey reports we recently found and counted how many multi-questions they asked, based on the plots their reports presented. The surveys are Kilpatrick Townsend CyberSec 2016, King Wood AustraliaDirs 2015, Littler Mendelson Employer 2018, and Morrison Foerster ConsumerProd 2018. In that order they have 7 multi-questions in 24 non-Appendix pages, 4 in 36 pages, 8 in 28 pages and 4 in 16 pages. Accordingly, results from at least 21 multi-questions appeared in 104 pages. Bear in mind that each report has a cover and a back page that have no plots and almost always other pages without plots so the total number of survey questions asked is always less than the number of report pages.
While multi-questions certainly allow more nuanced answers than “Pick the most important…” questions, for example, and create much more data, those more complicated pools of data challenge the survey analyst regarding how best to interpret and present it.
A number of analytic approaches enable an analyst to describe the results, to glean from the selection patterns deeper insights, and to depict them graphically. We will explore those techniques.