Being sure about confidence intervals

Using our data, we we can be 95\% confident that the true change in practicing lawyers for adding or subtracting one more student in a top-100 law school in the state lies between 2.5 and 7.8 lawyers. For a given confidence level (such as 95%), a narrower interval indicates a more precise estimate, whereas a broader interval indicates less precision. When the high boundary of the confidence interval is multiples of the low boundary, we are less sure of the association of the predictor and the response variable.

Note something else: because the confidence interval for percentage of the state’s population living in an urban area contains zero, a change in it is insufficiently related to practicing lawyer counts, statistically speaking, holding the other predictors constant. In other words, if the change might be zero, there is no statistically meaningful effect by that predictor. On the other hand, the law school enrollment interval not straddling zero, that predictor has a statistically significant p-value.

A confidence interval is an interval of good estimates of the unknown true population parameter.

Here is a plot with confidence intervals around the best-fit line of the Less500 predictor (companies with fewer than 500 employees). The intervals show as the shaded portions above and below the line. You can be 95 percent confident that the vertical range contains the true number of private practice lawyers for a state with the corresponding predictor value on the horizontal axis. If the predictor is indeed associated with the response variable, the more data a plot has in an area, the narrower the confidence interval, as you are more and more sure of the estimate.

One more aspect. We should explain how statisticians use the terms population and sample. The entire set of what you would like to count is the population; the portion of the population that you obtain is a sample from that population. So, for example, all the 45 associates in a firm comprise a population; the 15 selected at random to take a survey would be a sample from the population.

Statistics offers an impressive toolbox for making inferences from a partial sample to the entire population — and stating how likely those inferences are correct.

If we repeatedly sampled from the larger population (different mixes of 15 associates each time), the confidence intervals would contain the true population mean of whatever we are estimating from the linear regression. In other words, there is a 95% chance of selecting a sample such that the 95% confidence intervals calculated from that sample contain the correct mean for the response variable.

The confidence level does not express the chance that repeated sample estimates will fall into the confidence interval. Nor does it give the probability that the unknown mean for the response variable lies within the confidence interval.

Imputation for missing data when machine learning

Most collections of data have holes, missing data points. You don’t have the law school graduation year for this associate or the number of matters worked on for that associate. When you include those associate’s information in your regression modelling, the software may drop the associate totally because one piece is missing. You don’t want that to happen because then you have also lost the remaining, valid data of the associate.

Likewise, to shift examples, if you are studying your firm’s fees charged for reviewing securities law filings and you have completed 65 such matters over the past few years, but 10 of them are missing a number for revenue of the client, you actually have shrunk the analyzable set to only 55 matters.

Wanting to know what’s missing, as always with analyses a picture is invaluable. Here is a map of a data set with 500 observations that has some values missing in some of its 17 variables. A light, vertical line means that the observation on the horizontal axis had no value for that variable.

Make sure that no pattern explains missing data, such as if all the corporate department lawyers have no evaluation scores. But let’s assume that your data is missing at random, not for some identifiable reason like the Chicago office did not turn in its response sheet.

To counter the clobbering of good data caused by absent data, analysts resort to a range of methods to plug-in plausible figures and thereby save the remaining data. These methods, called imputation, are an important step when you prepare data for analysis.

The simplest method plugs in the average or median of all the values for that variable. Doing this, the average or median year of law school graduation would be inserted for the unknown year of an associate. Many other methods are available, with increasing amounts of calculations needed but with imputed values that are likely to be closer to the actual unavailable data. For example, you can run a regression model based on what you know and predict the value(s) you don’t know. For more on data imputation, see my article, Rees W. Morrison, “Missing in Action: Impute Intelligently before Deciding Based on Data”, LegalTechnology News April 2017.

Infographs and quantifying their components in law firm survey reports

Here are the final two infographs in the set that produces our analytics. On the left below, Squire Sanders Retail 2013 [pg. 24] includes a modest infograph, but at least the firm identifies it as such.

Early on its report, HoganLovells CrossBorder 2014 has two pages of infographs.  Below is the relevant portion of the first of those pages [pg. 8].

The data from all the above counting or estimating appears in the table below.

Infographs in survey reports: advantages and disadvantages

Infographs push law-firm survey reports as far as they currently go in terms of data visualization. Only a handful of them have been located, but they are enough to start an analysis.

Baker McKenzie Brexit 2017 [pg. 1] put its entire report into a single page of an infograph, as shown above.

McDonald Hopkins Business 2017 summarized its survey in early 2017 on business confidence. A snippet of the infograph the law firm produced — but did not include in the report itself — appears immediately above.

The list that follows pulls together a number of the reasons a law firm might want to invest in an infograph, and a number of reasons why it might take a pass.


  1. Links pieces of data collected by a survey.
  2. Tells a story.
  3. Helps readers understand a more complicated message than stand-alone plots.
  4. Attractive and eye-catching.
  5.  Trendy and exploits sophisticated software.
  6.  Produces a new asset for the firm to use in multiple ways.


  1. Complicated and mentally fatiguing for readers.
  2. Requires specialists in layout, design and communication, possibly commercial software.
  3. Different than just assembling several plots on a page.
  4. Requires more sophisticated thinking and planning of the message.
  5. May not justify the investment of time and resources.


Number and Percentage of Respondents Choosing Each Selection

Let’s remind ourselves of what we are calling “multi-questions.” The next plot, from a research-survey report (Kilpatrick Townsend CyberSec 2016 [pg. 7]) illustrates one. The plot derives from a multiple-choice question that listed seven selections and where “More than one choice permitted” applied. The plot gives the percentage of the 605 respondents who chose each selection.

You can spot such multi-questions because the percentages in the plot add up to more than one hundred. Here they total 237% which means an average of 2.37 selections per respondent.

Now, about presenting the results of multi-questions. Other than prose, the simplest description of the distribution of responses to a multi-choice question is a table. A table succinctly tells how many respondents chose each selection. From the data set we have been using and the question’s nine selections, the total number of roles selected was 318 from 91 respondents. A maximum of 819 possible selections could have been made if each respondent had checked each selection. When you know the number of participants in your survey, you can add a column for percentages.

If a table is not sorted by a relevant column, like the table above is sorted on “Selected”, it is harder for readers to compare frequencies. Column charts use bar height to help with comparisons, as the plot below illustrates. We used the data in the table above and added the frequency of selection in each bar.

Average number of pages in reports by originating law firm’s geography

From the period 2013 through now, we have found 154 research surveys where a law firm conducted or sponsored the survey and a PDF report was published. That group includes 55 different law firms.

We categorized the firms according to five geographical bases: United States firms, United Kingdom firms, vereins, combinations of U.S. and U.K. firms (“USUK”), and the rest of the world (“RoW” — Australia, Canada, Israel, New Zealand, and South Africa). We thought we would find that the largest firms, either the vereins or the USUK firms, would write the longest reports. Our reasoning was that they could reach more participants and could analyze the more voluminous data more extensively (and perhaps add more marketing pages about themselves).

Quite true! As can be seen in the table below, the average number of pages and the median number of pages for the five geographical groupings of firms each stand at approximately the same number. How many surveys are included in each category is shown in the column entitled “Number”. Nevertheless, the two large classes of firms do indeed produce more pages of reports.

GeoBase Number AvgPages MedianPages
RoW 13 25.0 20.0
UK 41 24.1 20.0
US 78 22.5 19.0
USUK 17 30.2 22.0
Verein 5 27.6 28.0

We tested the difference between the average number of pages for the USUK reports and average pages for the US reports. We selected those two groups because they had the largest gap [30.2 versus 22.5].

A statistical test called the t-test looks at two averages and the dispersion of values that make up each average. It tells you how likely it is that the difference of those averages is statistically significant, meaning that if random samples of survey reports were taken repeatedly from law firms in each geography, less than 5% of the time a gap of that amount or more would show up. If that threshold is not met, you can’t say that the differences are due to anything other than chance. If the threshold is met, statistician say that the difference can be relied on, in that it is statistically significant. On our data, the t-test was 1.2 and the p-value is 0.24, much above the threshold of 0.05 for statistical significance. The swing between USUK average pages and US average pages may look material, but on the data available, we can’t conclude that something other than random variation accounts for it.

Unconventional treatments of area plots

Area plots are unusual in research survey reports, but when they do appear they often seduce graphic designers into creating unconventional varieties. Consider this plot from DLA Piper Compliance 2017 [pg. 6]. Its most prominent irregularity is that the circles are not arranged in order of size. This conjures up depictions of the planets of our solar system! Further, the colors chosen are not meaningful, but they are distracting.

The area of each circle is proportional to the percentage of one of the seven job titles. A more customary layout would present the circles in declining size from the left or in ascending size to the right. Shall we call this an imaginative array?

KL Gates GCDisruption 2018 [pg. 8] also makes poor use of the area technique: the three percentages are too similar for the eye to pick up differences from the area of the circles. Worse, the circles are not aligned at the bottom so that readers can better detect differences in their areas. The firm could have opted to present this simple data in prose or with a small table.

It is hard enough for most people to discern differences in the area of similar circles, let alone when the area is represented by an object as unfamiliar as proportional bottles. Nevertheless, Reed Smith Lifesciences 2015 [pg. 10] chose a visualization technique that did exactly that. Furthermore, the percentages at the top are washed out.

Morrison Foerster GCsup 2017 [pg. 10] also chose an area plot even though there is not much difference between the areas of these circles. Plus the dual levels are very complex to understand. Compounding both effects is a very elaborate explanation below the plot.

Icons strengthen understanding and remembering

Here we use the term icon for a visual element in a survey report that is intended to convey a concept. By this definition, an icon serves more than a decorative purpose; it should link to and strengthen a discussion, plot or topic and add to the reader’s understanding and recall.

In Paul Hastings China 2013 [pg. 21], the firm chose icons to represent regulatory approval, the work done to integrate companies after an acquisition, and the due-diligence diving that precedes a potential acquisition. The visual representations of these three concepts, which are made clearer by the explanatory terms to the left, complement the text, a method that appeals to different cognitive styles of readers. Some people absorb information better by reading, others absorb information better through pictures. 1 Additionally, people store concepts in memory differently depending on the style of presentation.

HoganLovells Brexometer 2017 [pg. 9] turns to the well-known images of happiness and sadness. They stand atop a small table of survey results. Ever since the movie, Forest Gump, these stylized faces have become ubiquitous.

On page 9 of Pinsent Mason Infratech 2017, a closed-circuit TV camera films the lower right-hand corner of the page. It is not clear what that icon conveys, but a careful examination of the report shows that it is dappled with meaningful icons. You see a helicopter and an airplane on page 7, a stylized column chart with a trend arrow on page 14, light bulbs throughout the report that signify insights, a trio of humanoids on page 22, eight pages later four coins in a stack, an arrow in a target on page 34, and a trophy on page 38. Icon count no more.


  1. The day will come when reports include audio material for those who are aurally inclined.

Challenges choosing categories, e.g., for revenue demographics

When a law firm invites its contacts to to take a survey, those people who accept probably form an irregular group in terms of the distribution of their corporate revenue.  Their revenue can range by the happenstance of self-selection and how the list was compiled from negligible revenue to many billions of dollars. When the firm’s report describes the revenue characteristics of the group, the firm must decide what ranges of revenue to use.

The firm might slice the revenue categories to put in them roughly equal numbers of participants. Doing this usually means that the largest category spans a wide range of revenue — “$3 billion and up” — whereas the smallest category tends to be narrow — “$0 to 100 million.” Such an imbalance of ranges results from the real-world distribution of companies by revenue: lots and lots of smaller companies and a scattering of huge ones (the distribution is long-tailed to the right). Stated differently, the corporate revenue pyramid displays a very, very broad base.

Alternatively, a law firm might choose to set the revenue ranges by some specific range values, perhaps “\$0-1 billion, \$1-2 billion, \$2-3 billion” and so on. The categories may make sense a priori, but binning revenue this way can result in very uneven numbers of participants in one or more of the categories depending on what categories are chosen, how narrow they are, and the vagaries of who responds.

Davies Ward Barometer (2010) [pg. 10] explained the corporate revenue ranges of its respondents in words. These are unusual ranges. The distribution skews toward remarkably small companies. Note from the last bullet that almost one out of three survey respondents “are not sure of their organization’s annual revenue.” Perhaps they do not want to disclose that revenue, as they work for a privately-held company. Or perhaps the organization has no “revenue,” but has a budget allocation as a government agency.

With a third approach, a firm fits its revenue categories to its available data set so that plots look attractive. You can guess when a firm selects its revenue categories to fit its data set. Consider the plot below from DLA Piper’s compliance survey (2017) [pg. 26]. The largest companies in the first category reported less than $10 million in revenue; the next category included firms with up to 10 times more revenue, but about the same percentage of respondents; the third revenue category again spanned companies with up to ten times more revenue, topping out at $1 billion, but close to the preceding percentages. Then we see a small category with a narrow range of $400 million followed by the two on the right with half the percentages of the left three. It appears that someone tried various revenue categories to find a combination that looks sort of good in a graphic.

The fullest way to describe the revenue attributes for participants turns to a scatter plot. From such a plot, which shows every data point, readers can draw their own conclusions about the distribution of revenue.

Drop-down lists for multiple-choice questions

With a survey question in the style of “In the coming year, how will spending on cybersecurity at your law department likely change?”,  it is easiest for the surveyor to make respondents choose from set of answers (we refer to them as selections devised by the firm. The selections for that question might be “Increase more than 10%”, “Increase 6-10%”, “Increase 1-5%,” and on down to “Decrease more than 10%”.

It is easier for the firm to have pre-defined selections than to give respondents free rein to type in their answer as they see fit. The analyst will endure much pre-processing to clean the inevitable mishmash of styles respondents come up with — even if the questionnaire is laden with explicit instructions. People ignore guides such as “Only enter numerals, so no percent signs or ‘percent'”; do not write ranges such as “3-5” or “4 to 6”, do not add “approx” or “~”. No matter how clear you are, respondents will often jot in whatever they want.

Page 30 of Winston & Strawn’s 2013 report on risk displays the results of a question: “Your parent company’s annual revenues/turnover for the most recent fiscal year are:” Given the plot’s six categories of revenue, the questionnaire likely laid out those categories to choose from. Imagine two rows of three selections each. The selections were likely in order from the largest revenue category to the smallest and there was probably a check box or circle to click on next to each one. See how the plot below displays the data.

With only six selections, the questionnaire can efficiently lay them all out for consideration. Instead of displaying all of the answer choices beneath the question, a drop-down question shows a text box under the question and invites respondents to click on a down arrow to review a scrollable list. They pick their (single) answer from that list and the answer is filled in for them.  Drop-down questions tend to appear when there is a large list of selections, such as states of the United States or countries in Europe or months of the year. Almost all drop-downs have the capability to complete a partial entry, so that if you put in “N” Nebraska shows up and populates (fills in) the text-entry box.

Specialists in survey design recommend that drop-down questions be used sparingly. The examples above (years, months, states, countries) make sense because they have numerous choices and respondents are not evaluating which choice is best: one answer and only one answer is the right one. Demographic questions are ripe for drop-down treatment.

For most multiple choice questions, especially those with concepts and jargon, showing all choices at the same time gives respondents context as they answer the question. What you hope they are doing is considering the selections as a group and evaluating which one (or more) they favor in comparison to the others.

Winston & Strawn might have elected to use a drop-down list for the corporate revenue question. That list could have had many more revenue categories than six, which would have collected revenue more precisely, yet enforced a consistent style for the answers. On the other hand, that arrangement would have pushed respondents to think about their company’s revenue, or even to research it, and it would have taken more time for them then to spot the corresponding category from the drop-down list. Finer categories may also conflict with some respondents desire to remain anonymous or not to disclose sensitive information. Someone working at a privately-held company might be willing to click on the broad “1-5 billion” choice but not want to disclose a more specific revenue number.