Survey results are data that legal managers often encounter, but they need to be savvy about how much reliance to put on the data. When sophisticated survey results are published, the report typically includes a statement of the margin of error. Let’s think of a scenario. If the ABA invited its members by an email to approve or disapprove a nominee for the Supreme Court and 1,000 or so responded, the margin of error would be plus or minus three percent. So, if 60% approved the nominee, the “real” response from that subset (the sample) if all of the members (the population of ABA members) could be as high as 63% or as low as 57%. As explained in the NY Times, Oct. 6, 2016 at A18, the stated margin of error explains sampling variation: “error that occurs because surveys are based on only a subset of the full population of [ABA members].”
But the stated margin of error on a survey misses other important sources of error. “Frame error occurs when there is a mismatch between the people who are possibly included in the poll (the sampling frame) and the true target population.” With the ABA example, members who had not provided a usable email address would fall outside the frame of potential respondents.
A second form of survey error arises from nonresponders. That is when “the likelihood of responding to a survey is systematically related to how one would have answered the survey.” Again on the hypothetical ABA study, members who do not appear in court might not bother to respond at a higher rate than the rest of the members, and they might somewhat consistently (that is the meaning of “systematically” in the quote) have favored a judge who has been in the public eye regardless of merit. This would illustrate non-response bias.
Third, error other than margin of error may result from the analysis of the data. This is a morass of tough decisions and potential mistakes.
Fourth, the wording of the question or the choices permitted to the respondents (as in drop-down menus) may skew the accuracy of the results.
We could go on with other survey traps, but the key point here is that survey data in the legal industry should trigger careful examination of the survey’s methodology.