NAICS classification of industries would help surveys four ways

If only there were a standard way to describe survey participants by industry … There is! Law firms could identify, analyze, and report on their participants by the North American Industry Classification System (NAICS) categories. This system has moved beyond the venerable SIC (Standard Industrial Code) categories. The NAICS offers a range of two-digit classifications that map well to the extant proliferation of industry/sector designations seen in law firm reports. Those classification together with the three- and four-digit elaborations on them easily suffice for law-firm research surveys.

If NAICS codes became the convention for law firm research surveys, at least four benefits would follow.

Mash-up data. For data analysts, “mash-up” describes the process of melding two sets of data. If firms used the NAICS, other data would then be available for analysis. Longitudinal data sets, meaning those maintained over a period of time, that the U.S. government has collected by NAICS code can supplement information about the number of businesses in the industry, more detail about those businesses, the number of employees in the businesses, and so forth. Everyone would benefit from richer, more insightful analyses after various mash-ups.

Consistency among surveys. If law firms adopted this standard classification system, readers of their reports and researchers would be much more able to compare results by industries. In the current disorder, and so long as each firm defines its industries idiosyncratically, comparisons and meta-analyses become much harder to carry out, if not impossible.

Improving the representativeness of the sample data. Because the NAICS data sets provide law firms with reliable counts of companies by industry, they could deploy techniques to make their convenience samples more representative of the actual distribution of U.S. businesses. One method of doing this, which we explain elsewhere, is called “raking.” As sample data is transformed to closely resemble population data, deeper statistical analyses become available.

Impute missing values. “Imputation” is the term statisticians use for filling in missing values. If a law firm has data about its participants by their NAICS code plus other information such as revenue, the firm could impute the number of employees of that company. An explanation of that methodology to supplement data can be found elsewhere, but it would be available to a firm so long as the industry coding conforms to the NAICS. For example, a firm that collects revenue, industry code, and state can even more accurately impute a number for employees. Fuller data sets enable better analyses.

Average percentages of participants by industry

A truism: what organizations do varies wildly. Aside from government entitities and not-for-profits, even among the welter of profit-seeking organizations (companies and partnerships) the idiosyncracies of their activities often make it difficult to pigeon-hole the them as part of a single “industry”. Yet in the face of such “blooming buzzing confusion,” law-firm surveys request their participants to self-describe their unitary industry (aka sector). Making the mishmash worse, law firms lack standardized names for industries 1 so the terminology they present in their reports’ demographic profiles wanders all over.

To sample this dispersion of industry terms I looked at 14 surveys that were released in 2017. 2

I chose that set in part to bound the research and in part because the latest surveys might reflect whatever standardization of industry names has diffused through those firms. 3

As discussed before, disclosures regarding industries of participants varies from none at all to careful breakdowns and percentages. In this group, only four listed their participants’ industries with their respective percentages. To get an average percent by industry, I converted several of the industry names in the report to standardized names. The first column of the table contains the industry name used in a report (and to save space sometimes from two reports) while the second column contains the industry to which I assigned it. Other report names matched my standard names exactly or closely.

A profile of participants by industry not only credentializes the report but also allows firms to analyze their data by industry. Readers also welcome industry designations so that they can focus on data pertaining to their own industry, if the analysis proceeds to that level.

The plot shows the 12 standard industry names where at least two reports included both the industry (as standardized) and a percentage of all respondents in it. As can be seen from the number labels, most of the industries had only two values. The dots align on the bottom axis with the average of the two or more values for each industry. Thus, Finance appeared in various forms in all four surveys and averaged 28% of all their respondents.

Interpreted broadly on this small data set, the more often surveys identify a particular industry, the higher its percentage of respondents. This makes sense. Larger industries are more likely to be recognized, identified, and represented.

 

Notes:

  1. We pass over whether any enterprise can realistically be summarized in a single term.
  2. CARLTON FIELDS JORDAN BURT, THE 2017 CARLTON FIELDS CLASS ACTION SURVEY: BEST PRACTICES IN REDUCING COST AND MANAGING RISK IN CLASS ACTION LITIGATION (2017); CLIFFORD CHANCE, INSIGHTS INTO ASIA PACIFIC M&A (2016); DLA PIPER, DLA PIPER’S 2017 COMPLIANCE & RISK REPORT: COMPLIANCE GROWS UP INCREASING BUDGETS AND BOARD ACCESS POINT TOWARD GREATER PROMINENCE, INDEPENDENCE (2017); HAYNES AND BOONE, HAYNES AND BOONE, LLP BORROWING BASE REDETERMINATIONS SURVEY: FALL 2017 (2017); HOGAN LOVELLS, BREXOMETER (2017); K&L GATES, GENERAL COUNSEL IN THE AGE OF DISRUPTION (2017); LITTLER MENDELSON, THE LITTLER ANNUAL EMPLOYER SURVEY (2017); MORRISON \& FOERSTER, M+A SEMI-ANNUAL LEADERS SURVEY (2017); NORTON ROSE FULBRIGHT, 2017 LITIGATION TRENDS ANNUAL SURVEY: PERSPECTIVES FROM CORPORATE COUNSEL (2017); NORTON ROSE FULBRIGHT, REPUTATIONAL RISK AUSTRALIA (2017); PROSKAUER, VALUE INSIGHTS: DELIVERING VALUE IN LABOR AND EMPLOYMENT LAW — A SURVEY OF IN-HOUSE DECISION-MAKERS ON LABOR AND EMPLOYMENT MATTERS (2017); ROPES & GRAY, RISKY BUSINESS: MITIGATING EXPOSURE THROUGH COMPREHENSIVE RISK MANAGEMENT (2017); SEYFARTH SHAW, 2017 REAL ESTATE MARKET SENTIMENT SURVEY (2017); and WHITE & CASE, MINING & METALS 2017: A TENTATIVE RETURN TO FORM (2017).
  3. More surveys from 2017 almost certainly exist, and I would welcome hearing from any reader about any not listed in the footnote. I also omitted one 2017 survey because it had no industry data and two other surveys by another firm because I included a third from that firm.

Balancing survey respondents across industries and geographies

When most law firms conduct a research survey, they primarily hope to get enough respondents so that the results are defensible and generalizable. That is to say, they want enough data from desirable respondents to be able to say that their findings make sense and can be extrapolated beyond their particular group of participants.

Ropes & Gray made a very different decision in its 2017 survey on risk management practices in companies. Working with a research group, the firm deliberately balanced the number of participants by five named industries plus “Other” and across four geographical regions. As can be seen in the table below, adapted from page 5 of the report, out of the 300 total respondents, 100 came from each of America and EMEA, while 70 came from Pacific Asia and 30 from Latin America. Moreover, each industry had exactly 50 participants.

 AmericaEMEAPacific AsiaLatin AmericaTotal
Banking161712550
Private Equity171612550
Asset Management161811550
Life Sciences & Healthcare171711550
Technology171612550
Other171612550
Total1001007030300

The very brief description of the survey’s methodology does not explain why the firm chose those industries, those geographies, or the balanced participant numbers within them. Nor does it delve into how FT Remark, the research firm that assisted Ropes \& Gray, obtained the desired number of respondents.

One reason for the geographic distribution may have been that it proved difficult to obtain equal numbers of respondents for each pair of industry and geography. It may also be that the firm feels that this geographical weighting in some way more accurately represents companies and their risk management approaches around the globe. Many other questions arise regarding the decisions underlying this symmetric data set.

We will close by noting that if a law firm sets its goal to proportionally balance the number of responses by one or more criteria, revenue of the company being another possible parameter, it significantly increases the effort to locate and persuade the requisite number of respondents.