Infographs and quantifying their components in law firm survey reports

Here are the final two infographs in the set that produces our analytics. On the left below, Squire Sanders Retail 2013 [pg. 24] includes a modest infograph, but at least the firm identifies it as such.

Early on its report, HoganLovells CrossBorder 2014 has two pages of infographs.  Below is the relevant portion of the first of those pages [pg. 8].

The data from all the above counting or estimating appears in the table below.

Infographs in survey reports: advantages and disadvantages

Infographs push law-firm survey reports as far as they currently go in terms of data visualization. Only a handful of them have been located, but they are enough to start an analysis.

Baker McKenzie Brexit 2017 [pg. 1] put its entire report into a single page of an infograph, as shown above.

McDonald Hopkins Business 2017 summarized its survey in early 2017 on business confidence. A snippet of the infograph the law firm produced — but did not include in the report itself — appears immediately above.

The list that follows pulls together a number of the reasons a law firm might want to invest in an infograph, and a number of reasons why it might take a pass.

Advantages

  1. Links pieces of data collected by a survey.
  2. Tells a story.
  3. Helps readers understand a more complicated message than stand-alone plots.
  4. Attractive and eye-catching.
  5.  Trendy and exploits sophisticated software.
  6.  Produces a new asset for the firm to use in multiple ways.

Drawbacks

  1. Complicated and mentally fatiguing for readers.
  2. Requires specialists in layout, design and communication, possibly commercial software.
  3. Different than just assembling several plots on a page.
  4. Requires more sophisticated thinking and planning of the message.
  5. May not justify the investment of time and resources.

 

Number and Percentage of Respondents Choosing Each Selection

Let’s remind ourselves of what we are calling “multi-questions.” The next plot, from a research-survey report (Kilpatrick Townsend CyberSec 2016 [pg. 7]) illustrates one. The plot derives from a multiple-choice question that listed seven selections and where “More than one choice permitted” applied. The plot gives the percentage of the 605 respondents who chose each selection.

You can spot such multi-questions because the percentages in the plot add up to more than one hundred. Here they total 237% which means an average of 2.37 selections per respondent.

Now, about presenting the results of multi-questions. Other than prose, the simplest description of the distribution of responses to a multi-choice question is a table. A table succinctly tells how many respondents chose each selection. From the data set we have been using and the question’s nine selections, the total number of roles selected was 318 from 91 respondents. A maximum of 819 possible selections could have been made if each respondent had checked each selection. When you know the number of participants in your survey, you can add a column for percentages.

If a table is not sorted by a relevant column, like the table above is sorted on “Selected”, it is harder for readers to compare frequencies. Column charts use bar height to help with comparisons, as the plot below illustrates. We used the data in the table above and added the frequency of selection in each bar.

Average number of pages in reports by originating law firm’s geography

From the period 2013 through now, we have found 154 research surveys where a law firm conducted or sponsored the survey and a PDF report was published. That group includes 55 different law firms.

We categorized the firms according to five geographical bases: United States firms, United Kingdom firms, vereins, combinations of U.S. and U.K. firms (“USUK”), and the rest of the world (“RoW” — Australia, Canada, Israel, New Zealand, and South Africa). We thought we would find that the largest firms, either the vereins or the USUK firms, would write the longest reports. Our reasoning was that they could reach more participants and could analyze the more voluminous data more extensively (and perhaps add more marketing pages about themselves).

Quite true! As can be seen in the table below, the average number of pages and the median number of pages for the five geographical groupings of firms each stand at approximately the same number. How many surveys are included in each category is shown in the column entitled “Number”. Nevertheless, the two large classes of firms do indeed produce more pages of reports.

GeoBase Number AvgPages MedianPages
RoW 13 25.0 20.0
UK 41 24.1 20.0
US 78 22.5 19.0
USUK 17 30.2 22.0
Verein 5 27.6 28.0

We tested the difference between the average number of pages for the USUK reports and average pages for the US reports. We selected those two groups because they had the largest gap [30.2 versus 22.5].

A statistical test called the t-test looks at two averages and the dispersion of values that make up each average. It tells you how likely it is that the difference of those averages is statistically significant, meaning that if random samples of survey reports were taken repeatedly from law firms in each geography, less than 5% of the time a gap of that amount or more would show up. If that threshold is not met, you can’t say that the differences are due to anything other than chance. If the threshold is met, statistician say that the difference can be relied on, in that it is statistically significant. On our data, the t-test was 1.2 and the p-value is 0.24, much above the threshold of 0.05 for statistical significance. The swing between USUK average pages and US average pages may look material, but on the data available, we can’t conclude that something other than random variation accounts for it.

Unconventional treatments of area plots

Area plots are unusual in research survey reports, but when they do appear they often seduce graphic designers into creating unconventional varieties. Consider this plot from DLA Piper Compliance 2017 [pg. 6]. Its most prominent irregularity is that the circles are not arranged in order of size. This conjures up depictions of the planets of our solar system! Further, the colors chosen are not meaningful, but they are distracting.

The area of each circle is proportional to the percentage of one of the seven job titles. A more customary layout would present the circles in declining size from the left or in ascending size to the right. Shall we call this an imaginative array?

KL Gates GCDisruption 2018 [pg. 8] also makes poor use of the area technique: the three percentages are too similar for the eye to pick up differences from the area of the circles. Worse, the circles are not aligned at the bottom so that readers can better detect differences in their areas. The firm could have opted to present this simple data in prose or with a small table.

It is hard enough for most people to discern differences in the area of similar circles, let alone when the area is represented by an object as unfamiliar as proportional bottles. Nevertheless, Reed Smith Lifesciences 2015 [pg. 10] chose a visualization technique that did exactly that. Furthermore, the percentages at the top are washed out.

Morrison Foerster GCsup 2017 [pg. 10] also chose an area plot even though there is not much difference between the areas of these circles. Plus the dual levels are very complex to understand. Compounding both effects is a very elaborate explanation below the plot.

Icons strengthen understanding and remembering

Here we use the term icon for a visual element in a survey report that is intended to convey a concept. By this definition, an icon serves more than a decorative purpose; it should link to and strengthen a discussion, plot or topic and add to the reader’s understanding and recall.

In Paul Hastings China 2013 [pg. 21], the firm chose icons to represent regulatory approval, the work done to integrate companies after an acquisition, and the due-diligence diving that precedes a potential acquisition. The visual representations of these three concepts, which are made clearer by the explanatory terms to the left, complement the text, a method that appeals to different cognitive styles of readers. Some people absorb information better by reading, others absorb information better through pictures. 1 Additionally, people store concepts in memory differently depending on the style of presentation.

HoganLovells Brexometer 2017 [pg. 9] turns to the well-known images of happiness and sadness. They stand atop a small table of survey results. Ever since the movie, Forest Gump, these stylized faces have become ubiquitous.


On page 9 of Pinsent Mason Infratech 2017, a closed-circuit TV camera films the lower right-hand corner of the page. It is not clear what that icon conveys, but a careful examination of the report shows that it is dappled with meaningful icons. You see a helicopter and an airplane on page 7, a stylized column chart with a trend arrow on page 14, light bulbs throughout the report that signify insights, a trio of humanoids on page 22, eight pages later four coins in a stack, an arrow in a target on page 34, and a trophy on page 38. Icon count no more.

Notes:

  1. The day will come when reports include audio material for those who are aurally inclined.

Challenges choosing categories, e.g., for revenue demographics

When a law firm invites its contacts to to take a survey, those people who accept probably form an irregular group in terms of the distribution of their corporate revenue.  Their revenue can range by the happenstance of self-selection and how the list was compiled from negligible revenue to many billions of dollars. When the firm’s report describes the revenue characteristics of the group, the firm must decide what ranges of revenue to use.

The firm might slice the revenue categories to put in them roughly equal numbers of participants. Doing this usually means that the largest category spans a wide range of revenue — “$3 billion and up” — whereas the smallest category tends to be narrow — “$0 to 100 million.” Such an imbalance of ranges results from the real-world distribution of companies by revenue: lots and lots of smaller companies and a scattering of huge ones (the distribution is long-tailed to the right). Stated differently, the corporate revenue pyramid displays a very, very broad base.

Alternatively, a law firm might choose to set the revenue ranges by some specific range values, perhaps “\$0-1 billion, \$1-2 billion, \$2-3 billion” and so on. The categories may make sense a priori, but binning revenue this way can result in very uneven numbers of participants in one or more of the categories depending on what categories are chosen, how narrow they are, and the vagaries of who responds.

Davies Ward Barometer (2010) [pg. 10] explained the corporate revenue ranges of its respondents in words. These are unusual ranges. The distribution skews toward remarkably small companies. Note from the last bullet that almost one out of three survey respondents “are not sure of their organization’s annual revenue.” Perhaps they do not want to disclose that revenue, as they work for a privately-held company. Or perhaps the organization has no “revenue,” but has a budget allocation as a government agency.

With a third approach, a firm fits its revenue categories to its available data set so that plots look attractive. You can guess when a firm selects its revenue categories to fit its data set. Consider the plot below from DLA Piper’s compliance survey (2017) [pg. 26]. The largest companies in the first category reported less than $10 million in revenue; the next category included firms with up to 10 times more revenue, but about the same percentage of respondents; the third revenue category again spanned companies with up to ten times more revenue, topping out at $1 billion, but close to the preceding percentages. Then we see a small category with a narrow range of $400 million followed by the two on the right with half the percentages of the left three. It appears that someone tried various revenue categories to find a combination that looks sort of good in a graphic.

The fullest way to describe the revenue attributes for participants turns to a scatter plot. From such a plot, which shows every data point, readers can draw their own conclusions about the distribution of revenue.

Drop-down lists for multiple-choice questions

With a survey question in the style of “In the coming year, how will spending on cybersecurity at your law department likely change?”,  it is easiest for the surveyor to make respondents choose from set of answers (we refer to them as selections devised by the firm. The selections for that question might be “Increase more than 10%”, “Increase 6-10%”, “Increase 1-5%,” and on down to “Decrease more than 10%”.

It is easier for the firm to have pre-defined selections than to give respondents free rein to type in their answer as they see fit. The analyst will endure much pre-processing to clean the inevitable mishmash of styles respondents come up with — even if the questionnaire is laden with explicit instructions. People ignore guides such as “Only enter numerals, so no percent signs or ‘percent'”; do not write ranges such as “3-5” or “4 to 6”, do not add “approx” or “~”. No matter how clear you are, respondents will often jot in whatever they want.

Page 30 of Winston & Strawn’s 2013 report on risk displays the results of a question: “Your parent company’s annual revenues/turnover for the most recent fiscal year are:” Given the plot’s six categories of revenue, the questionnaire likely laid out those categories to choose from. Imagine two rows of three selections each. The selections were likely in order from the largest revenue category to the smallest and there was probably a check box or circle to click on next to each one. See how the plot below displays the data.

With only six selections, the questionnaire can efficiently lay them all out for consideration. Instead of displaying all of the answer choices beneath the question, a drop-down question shows a text box under the question and invites respondents to click on a down arrow to review a scrollable list. They pick their (single) answer from that list and the answer is filled in for them.  Drop-down questions tend to appear when there is a large list of selections, such as states of the United States or countries in Europe or months of the year. Almost all drop-downs have the capability to complete a partial entry, so that if you put in “N” Nebraska shows up and populates (fills in) the text-entry box.

Specialists in survey design recommend that drop-down questions be used sparingly. The examples above (years, months, states, countries) make sense because they have numerous choices and respondents are not evaluating which choice is best: one answer and only one answer is the right one. Demographic questions are ripe for drop-down treatment.

For most multiple choice questions, especially those with concepts and jargon, showing all choices at the same time gives respondents context as they answer the question. What you hope they are doing is considering the selections as a group and evaluating which one (or more) they favor in comparison to the others.

Winston & Strawn might have elected to use a drop-down list for the corporate revenue question. That list could have had many more revenue categories than six, which would have collected revenue more precisely, yet enforced a consistent style for the answers. On the other hand, that arrangement would have pushed respondents to think about their company’s revenue, or even to research it, and it would have taken more time for them then to spot the corresponding category from the drop-down list. Finer categories may also conflict with some respondents desire to remain anonymous or not to disclose sensitive information. Someone working at a privately-held company might be willing to click on the broad “1-5 billion” choice but not want to disclose a more specific revenue number.

Challenge of clear selections if they cover complicated ideas

When you write non-numeric selections of a multiple-choice question, you want them to be different from each other and cover the likely choices as completely as possible. Yet at the same time you don’t want too many selections. You also would like them to be close to the same length. We have compiled other good practices.

The selections also need to be quickly and decisively understood by respondents. Respondents don’t want to puzzle over meanings and coverage of terms. Partly that means you need to cure ambiguities but partly it means to choose terms in selections carefully so that nearly everyone interprets them the same way at first reading.

We found an instructive example in one of the law-firm research surveys. Did the questions in the plot below achieve quick clarity? 1

I wonder whether most of the general counsel understand “Partner ecosystem”, let alone in the same way. Should there be a two notions joined as in “New sources of revenue and new business models”? Some companies might pursue revenue or a new business model, but not both. Likewise, why pair “Clean energy and environmental regulation”? They could be seen as two separate trends. The selection “Geopolitical shifts” feels so broad that it invites all kinds of interpretations by respondents.

This question challenged the survey designers with an impossible task. First they had to pick the important trends — and what happened to “Demographic changes”, “Big data”, “Urbanization” and “Taxes” to pick a few others that could have been included? Second, they had to describe those multifaceted, complex trends in only a few words. Third, those few words needed to fix a clear picture in lots of minds, or else the resulting data represents a blurred and varied clump of subjective impressions.

Notes:

  1. We do not know if the left-hand-side labels mirror the selections on the questionnaire. Some surveys have more detail, and even explanations, but the report gives an abbreviation of the selection.

Mandatory annual disclosure of number of lawyers and revenue by U.S. law firms

Why can’t the American Bar Association (or State Bars) require U.S.-based law firms above some modest-sized number of lawyers to report their fiscal year revenue along with a snapshot of the number of partners, associates, and support staff on the last day of the year?  The justification for that disclosure would be that clients, law school graduates or lawyers considering a job change, among others, would have comprehensive and reliable data on at least two key attributes of firms: size and revenue.

Yes, there are definitional issues, such as what does the term “partner” mean in the multi-tiered law firms of today and what makes up “revenue”.   Yes, there might be no way to confirm the accuracy of the self-reported numbers, but law firms that would have to comply have their books audited or reviewed by accountants, and the accountants could to attest to the reasonable accuracy of the four numbers.  Yes, I do not know what enforcement mechanisms might be available.  And yes, firms may fear that the initial data request slips down the proverbial slope to more and more.

Such concerns would need to be debated; they can be resolved.  If firms that have more than 30 lawyers fell under this mandate, then perhaps 1,200 to 1,500 law firms would each year turn in four numbers that they already know.  No work would be required except going to an online site and filling in the numbers.  The ABA or a third party could consolidate and publish that data and the legal industry would be greatly the beneficiary.