It seems likely that surveys with more participants would cover more demographic attributes and divide those attributes into more categories. Larger numbers of respondents would encourage more slicing and dicing. Or so I thought.
To test that hypothesis, I looked at 10 law-firm research surveys. My less-than-scientific method to pick them started with the last one alphabetically on my list by firm name and pored over the surveys in reverse order until I found 10 with usable data. The by-passed surveys either did not disclose their number of participants, gave very sketchy demographic information, or both (a few were in a series by the same firm). Having trapped the eligible surveys, I counted how many categories the report included for each of the four most common demographics — position of the respondent, revenue of the respondents’ companies, location of the companies, and industry (what I have called the “Big 4”).
The first of the two charts shows how the total number of categories in those four demographic attributes compares to the number of participants in the survey.Each red circle stands for one survey’s number of participants (on the bottom axis) and total Big 4 categories (on the left axis). The wavy blue line shows a non-linear trend line. Very non-linear, and not much of a pattern!
The second chart displays the same total of categories for the four most common demographics, plus the total number of categories in any other demographic attributes on the left axis. It reflects the same 10 surveys, so the bottom axis remains the same, but a greater range on the vertical axis because it includes counts from any other demographic attributes. The trend line here shows even less of a pattern than the squiggle of first plot!
Sigh. At least with this set of surveys, we can’t support a hypothesis that more participants means more demographic attributes. Perhaps if we broadened this particular inquiry to cover more surveys we might eventually distinguish a clearer relationship, but for the moment, none is apparent.
As with a previous post, to study revenue profiles I zeroed in on a set of 14 survey-research reports published by law firms in 2017. That post considers how fully and consistently the firms shared profile information on their participants’ industries. Here the focus turns to how that group of reports disclosed aggregated categories of their participants’ annual revenue.
To start with discouraging news, eight reports tell readers absolutely nothing about the revenue of their respondent population (one chose, oddly, to give three market cap categories). This tell-nothing decision by eight law firms is regrettable because readers of their reports are severely handicapped in judging how credible the report is and, more specifically, how well its findings apply to the reader’s organization. The omission of this profile (demographic) data also suggests that the firms did not analyze their findings by revenue categories.
Turning to the remaining six reports, four give only a single revenue indicator, such as “almost half the respondents reported revenue of greater than $1 billion.” In the plot below, those reports show only two bars: one for the amount of revenue stated and up and one slightly less than the amount stated and down. Thus, based on the single indicator, I created a binary categorization of revenue.The reports of Norton Fulbright and Hogan Lovells exemplify splendidly what survey reports should do. They broke their respondents into three or six revenue categories, respectively. As royalty, they deserve to have purple bars, compared to the yellow bars of the other firms. In the interests of full disclosure, I should note that I slightly modified some of the range data as given so that the the plot has more uniformity.
It is clear that the revenue categories applied by the firms that used categories cover an extremely wide range, from less than $99 million to more than $20 billion. Moreover, each firm conjured up its own category boundaries, with almost no standardization across the reports (except what I imposed). Three reports used “more than $ billion”, but that was the only shared category. As with demographic reporting on industries, everyone in the legal profession would gain if there were more disclosure of revenue demographics and more consistent use of similar bands.