Disclosure of revenue profiles by reports published in 2017

As with a previous post, to study revenue profiles I zeroed in on a set of 14 survey-research reports published by law firms in 2017. 1 That post considers how fully and consistently the firms shared profile information on their participants’ industries. Here the focus turns to how that group of reports disclosed aggregated categories of their participants’ annual revenue.

To start with discouraging news, eight reports tell readers absolutely nothing about the revenue of their respondent population (one chose, oddly, to give three market cap categories). This tell-nothing decision by eight law firms is regrettable because readers of their reports are severely handicapped in judging how credible the report is and, more specifically, how well its findings apply to the reader’s organization. The omission of this profile (demographic) data also suggests that the firms did not analyze their findings by revenue categories.

Turning to the remaining six reports, four give only a single revenue indicator, such as “almost half the respondents reported revenue of greater than $1 billion.” In the plot below, those reports show only two bars: one for the amount of revenue stated and up and one slightly less than the amount stated and down. Thus, based on the single indicator, I created a binary categorization of revenue.The reports of Norton Fulbright and Hogan Lovells exemplify splendidly what survey reports should do. They broke their respondents into three or six revenue categories, respectively. As royalty, they deserve to have purple bars, compared to the yellow bars of the other firms. In the interests of full disclosure, I should note that I slightly modified some of the range data as given so that the the plot has more uniformity.

It is clear that the revenue categories applied by the firms that used categories cover an extremely wide range, from less than $99 million to more than $20 billion. Moreover, each firm conjured up its own category boundaries, with almost no standardization across the reports (except what I imposed). Three reports used “more than $ billion”, but that was the only shared category. As with demographic reporting on industries, everyone in the legal profession would gain if there were more disclosure of revenue demographics and more consistent use of similar bands.

Notes:

  1. The post explains how the reports were chosen and which firms were represented in the data set.

Further observations on profile (demographic) data in survey reports

We looked further at how survey reports present information regarding their respondents’ profiles. To start a longer-term effort over a large set of law-firm research surveys, we selected four surveys at random: Carlton Fields Jordan Burt, “The 2012 Carlton Fields Class Action Survey: Best Practices in Reducing Cost and Managing Risk in Class Action Litigation” (2012); Hogan Lovells, “Risk and Return: Foreign Direct Investment and the Rule of Law” (2014); Littler Mendelson, “Littler Mendelson XMPT Survey Report” (2013); and Norton Rose, “Deloitte/Norton Rose ESOP survey report: Australian business’ experiences with Employee Share Options” (2014). From this spot review, we note several points.

Disclosure of four common profiles. Each survey offered at least some insight into the positions of the respondents and the companies’ geographical locations, annual revenue, and industry sector. None of the surveys went beyond simple counts (or percentages) of respondents or minimal descriptive statistics. One might hope to see scatter plots, perhaps of revenue, or cross tabulations, such as revenue figures broken out by industry. Nothing. One of the surveys, in fact, stated only the percentage of respondents in a handful of industries (out of 18 selections listed in the question) and did not give a complete breakdown.

Mixture of text and plots. These reports shared profile information sometimes only in text and other times with a mix of texts and plots. Here is an snippet from one report that used text.

Shortage of analysis with profile data. Disappointingly, only one of the four reports broke down the data the firm had collected by any profile attributes. That report includes tables and text about industry variance. None of them, for instance, showed how responses were influenced by the revenue size of the company or by the position of the respondent.

Happenstance participation. The Hogan Lovells survey appears to have tried to obtain participants somewhat in proportion to who makes foreign direct investments, the subject of the survey. The other three appear to have reached out to some undisclosed number of potential participants and treated the respondents who chose to take part as the data base. In the terminology of surveys, this was a convenience sample.  The profile information given did not suggest any efforts to balance the representation of participants.

Revenue plus rankings. Two of the surveys shared some data about participants’ corporate revenue and then added some information about how many of them fell into various strata of the Fortune 1000 or the Forbes 2000. This profile information argues particularly forcefully for how law firms use profile data less to carry out analyses than to boost credibility: “We have companies participating that are big and prestigious!”

Location of profile data. The reports comment on their distribution of participants at the start of the report and generally provide some more detail or plots at the end of the report in an appendix.