Few legal-industry surveys sample from a representative population

To respect and rely on the findings of a legal industry survey, legal managers should be able to find in the survey report the number of people who answered the survey (the sample, respondent number or sometimes just “N”), the number of people who were invited to answer the survey (the population), and how the surveyor developed that population of invitees.

Focus on that last disclosure, which basically concerns the representativeness of the survey population.  If a company that sells time and billing software to law firms writes to its customers and asks them “Do you find software technology valuable for your firm?,” no one should be surprised if the headline the vendor’s report boasts “Nine out of ten law firms find software technology valuable!”  Aside from binary choice of the question and the vendor’s blatant self-interest in promoting sales of software, the crucial skew in the results arises from the fact that the people invited to complete the survey hardly mirror people in law firms generally.  They have licensed or at least know about time and billing software.  The deck was stacked, the election was rigged.

Unfortunately, all too often vendor-sponsored surveys go out to invitees who have some connection with the vendor and therefore are hardly representative of law firm lawyers and staff as a whole.  The invitees will almost certainly be on the vendor’s contact list or its newsletter recipients or those who visit the vendor’s website and register.  Only sometimes will a vendor develop or rent a much larger mailing list and reach out to its names.  Even if they do, respondents will likely be self-selected because they use that kind of software or service or have some level of awareness of it.

Try a snowball survey so you get more participants in a client satisfaction or legal industry survey

Most law departments, when inviting their clients to complete a satisfaction survey, select recipients at or above a certain level, such as all “Managers,” or “everyone above comp level 15.”  It would be interesting and enlightening for a department to try a “snowball survey.”

Send the questionnaire form (or an email with its online equivalent) to a relatively few, high-level clients. Ask them to complete the form and also to forward the blank form to three colleagues who have worked recently with the law department (or forward the email invitation to those colleagues).  Each recipient, in turn, is also invited to extend the survey’s reach, and thus the snowball grows.

A service provider in the legal industry could adopt the same tactic: invite everyone you can reach to take a survey, but urge them to send it on to others they know who would have something to say about the survey’s topic.  Now, some surveyors may reject the snowball approach because they want to control who is possibly in the participation group.  But a broader-minded desire and one that is more objective would be to sample as many participants as possible and thereby gain a more accurate understanding of the entire population.