Time records of in-house counsel are suspect sources of data for analysis

Data scientists in law departments may have time records to analyze, but it is questionable whether they should “take the time” (pun intended).  The practice is riddled with methodological impurities as well as managerial challenges.

A fair number of larger law departments have their lawyers track time; some estimates place that number in the 20-30 percent range.  Many of those departments, however, use the hours-worked information only for internal purposes such as to manage workloads, understand the kinds of matters they handle, and evaluate performance.

Some departments that track internal lawyer time, however, charge time back to clients..  A chargeback system has drawbacks; it’s a hassle for the lawyers, it requires someone to handle the “billing,” it might dissuade a needy but impecunious client from getting legal advice, it’s just an intra-company transfer, and it might spawn inaccurate charges.  Perversely, if you bill clients for internal lawyer time, you dissuade them sometimes from seeking legal counsel or you lure them into forum-shopping (taking work to a lawyer who under-records time).  [We note that some utilities are required by state regulatory agencies to allocate their lawyers’ time between regulated and non-regulated activities so that the rate base is not skewed.]

Defenders of charging time to clients have their arguments: it gives some market discipline to what otherwise is a free, and possibly abused, service (but only if clients can contest charges); it adds another overseer of lawyer productivity – the client charged; it allows a law department to negotiate levels of service (though hours worked is as pernicious as hourly billing by law firms); and it gives quantitative backup to a plea to hire more lawyers (although no confirmation is possible).   Worse, some law departments require everyone to record eight hours a day, which makes a mockery of any claim that the data matches reality.

In the future for law (and now): machine learning software — my article

The term “machine learning” captures a teeming mélange of software algorithms that “learn” from data.  It is becoming a buzz word in the legal industry as law firms and law departments explore what machine learning tools can do.

As my first article in a series for LTN [Legal Technology News], I wrote about some of the fundamental concepts of machine learning.  You are invited to  read the full article by clicking on the link.  Let me know your thoughts, as there is a world of learning in the domain of law-related machine learning.

A makeover of an ILTA graph, explaining the improvements in visualization

Legal managers who create data-analysis graphs should strive to make those graphs effective communicators.  Let’s pause for a teaching moment.  I wrote a post about the 2016 ILTA/InsideLegal Technology Purchasing Survey and its question about areas of practice where respondents foresaw AI software penetrating.   

The plot in the upper right portion of page 13 that summarizes the answers to that question could be improved in several ways.


The bar colors are nothing but distracting eye-candy, since the colors do not convey any additional information.  If a couple of bars were colored to indicate something, that would be a different matter.

Second, it was good to add the percentages at the end of the bars, rather than force readers to look down at the horizontal axis and estimate them; however, if the graph states each bar’s percentage, the horizontal axis figures are unnecessary.  Even more, the vertical grey lines can be banished.

Third, most people care less about an alphabetical ordering of the bars than they do about comparisons among the applications on percentages.  It would have been more informative to order the bars in the conventional longest-at-the-top to shortest-at-the-bottom style.

As a kudo, it was good to put the application areas on the left rather than the bottom.  Almost always there is more room on the left than in the narrower bands at the bottom.


A makeover using the same data cures these problems and displays a few other visualization improvements.  The new plot removes the boundary lines around the plot, which gives a cleaner look.  It also enlarges the font on the percentages relative to the font on the applications, since those figures are likely to be the ones that readers care most about and want most emphasized.  Two final tweaks: the application names are on one line, and the axes have no “tick marks”, the tiny lines that mark the mid-point of an axis interval but that rarely add any value.

Extract-Load-Transform (ELT) cleans data later in the process than Extract-Transform Load (ETL)

A few days ago, I challenged a statement by David White, a director at AlixPartners,  in a column.  When I wrote to him to let him know, he responded right away, and taught me something.

He suggested “looking into the differences between the concepts of Extract-Transform-Load (ETL) and Extract-Load-Transform (ELT), especially with regard to big data technology such as Hadoop, MapReduce, Splunk and the like.   The process you describe [in the previous blog post] of needing to normalize data before analysis, such as trimming currency symbols, refers to the “transform” stage that traditional analytics system require be done upfront.  Newer technologies no longer require this to be done until much later, as part of the analytics process.  And often, not at all.  Therefore, we don’t have to worry that some of our values have dollar signs and some do not, or some dates are in one format and others in another.  We can extract and load all the data upfront, regardless and deal with these issues when building queries, if needed at all.  This feature is what allows these systems to perform analysis on multiple diverse formats of data, including both structured and unstructured data, at the same time, and on real time data feeds.  For example, Google doesn’t care about date formats or dollar signs when you run a query across millions of websites, yet returns accurate results regardless of the format of the original content.”

Excellent, and I appreciate David taking the time to explain why my challenge was misguided!

An introduction to software that identifies relevant documents in litigation

Software that analyzes text documents collected electronically in litigation discovery has developed a long way in the past few years. Variously referred to as “predictive coding,” “technology assisted review“ (TAR), and “computer-assisted review”, the software’s steps can be explained simply.

You train the software on a “seed set” of documents that humans have coded to be relevant or not and then you aim the software at a random sample of other documents that have also been coded for relevance, the “validation set.”

Assuming the software has been trained sufficiently well on the seed set, its effectiveness is judged by “recall,” which is the percentage of relevant documents in the random-sample validation set that the software accurately identified as such.  As pointed out in a recent article in LTN, Oct. 2016 at 40, by three lawyers at the law firm BuckleySandler, courts have paid attention to how the seed set was constructed, but haven’t paid as much attention to the accuracy of coding the validation set.

Once the level of recall satisfies both sides to the lawsuit, the software is unleashed on all the collected documents and it dutifully identifies those deemed by the algorithm to be relevant and thus producible.

Margin of error in legal industry surveys, and other traps

Survey results are data that legal managers often encounter, but they need to be savvy about how much reliance to put on the data.  When sophisticated survey results are published, the report typically includes a statement of the margin of error.  Let’s think of a scenario.  If the ABA invited its members by an email to approve or disapprove a nominee for the Supreme Court and 1,000 or so responded, the margin of error would be plus or minus three percent.  So, if 60% approved the nominee, the “real” response from that subset (the sample) if all of the members (the population of ABA members) could be as high as 63% or as low as 57%.  As explained in the NY Times, Oct. 6, 2016 at A18, the stated margin of error explains sampling variation: “error that occurs because surveys are based on only a subset of the full population of [ABA members].”

But the stated margin of error on a survey misses other important sources of error.  “Frame error occurs when there is a mismatch between the people who are possibly included in the poll (the sampling frame) and the true target population.”  With the ABA example, members who had not provided a usable email address would fall outside the frame of potential respondents.

A second form of survey error arises from nonresponders.  That is when “the likelihood of responding to a survey is systematically related to how one would have answered the survey.”  Again on the hypothetical ABA study, members who do not appear in court might not bother to respond at a higher rate than the rest of the members, and they might somewhat consistently (that is the meaning of “systematically” in the quote) have favored a judge who has been in the public eye regardless of merit.  This would illustrate non-response bias.

Third, error other than margin of error may result from the analysis of the data.  This is a morass of tough decisions and potential mistakes.

Fourth, the wording of the question or the choices permitted to the respondents (as in drop-down menus) may skew the accuracy of the results.

We could go on with other survey traps, but the key point here is that survey data in the legal industry should trigger careful examination of the survey’s methodology.

Reproducible research as a desiderata for legal data analysis

What is termed ‘reproducible research’ urges all data scientists to keep careful track of their data’s source and transformations.  Each step of the way from the original numbers – the headwaters – through each addition, subtraction, calculation or revision should be recorded so that another person could reproduce the final data set – the mouth of the river.  They should be able to evaluate the appropriateness of the complete stream of alterations and manipulations.

As to the provenance of data, the URL and date on which information was scraped from a web site would be crucial.  The publication, date and page of data obtained from print would be key.  How the original data was collected, such as by an export from an email system or survey needs to be spelled out.  And so on.

In the first instance, programmers approach reproducibility with the code itself, which tells another programmer in the same language what is going on, such as turning a character variable into a numeric variable or multiplying a group of numbers by something or choosing a subset of the data.   But often code alone can be cryptic, or the logic is not clear, or the reasons for certain choices that were made are murky and difficult to recreate.

Liberal commenting by the programmer can fill the gap to create a roadmap for others.  All programming languages have a simple method to say to the computer, “Ignore this line, it is a note to myself and others.”  Good programmers explain in comments what the following lines of code do, why the script is doing that, and any issues or decisions in play.   It is an excellent practice to write fulsome comments that would allow a non-programmer to follow the origin, transformations, and outputs of a data workflow.  Such comments, by the way, greatly help the programmer later when she returns to the now-forgotten analysis and has to reconstruct it.

Beyond spelling out the source of the data, the programming calls themselves, and ample comments, what is known as ‘literate programming’ gives guidelines for how the code should be divided up, indented, and how the supplementary annotations are added.

In the legal industry, data analysts should strive for reproducible research, transparency in every step of their work.

A benefit of sharing data analyses: thoughtful questions and interpretations

Any time a law department distributes an analysis of data to executives it may get questions or interpretations that the department’s legal managers had not thought about.  Likewise, when a managing partner of practice group head in a law firm circulates a data analysis, they should expect (indeed hope for) someone to ask about methodology or accuracy or the takeaways from the findings.

Not only is data transparency useful for marketing attorney effort and worth, it also stimulates thinking by the recipients – crowd-sourcing regarding legal management data.  General counsel should shine light into the black box of outside counsel costs – expose the numbers, explain variability, explicate the methodology.  Publish amounts spent by firm, by practice area, by business or staff unit – and where possible show trend data on all of them.  Present blended billing rates for key firms, and show the discounts negotiated from standard rates.   When executives understand more about costs they are probably less likely to carp; put negatively, ignorance coupled with suspicion breeds mistrust and criticism.

Or as Louis Brandeis memorably put it, “Publicity is justly commended as a remedy for social and industrial diseases. Sunlight is said to be the best of disinfectants; electric light the most efficient policeman.”  He might have added, “Candles promote thinking.”

Analytic software requires curating before it can proceed reliably

A column in Met. Corp. Counsel, Sept. 2016 at 21 by David White, a director at AlixPartners, starts with three paragraphs on the severity of anti-corruption risks to U.S. companies that do business abroad and the associated huge variety and volume of documents and records that might have to be analyzed to respond to regulators.  Data analytics to the rescue!

White continues: “Unlike their traditional counterparts, newer analytic systems based on big data technologies, predictive analytics and artificial intelligence are not bound by upfront data transformations and normalization requirements …” [emphasis added].

In my experience, analytical software inevitably requires the data fed to it to be formatted cleanly in ways that it can handle.  For example, dollars signs can’t be included, currencies need to be converted, dates need a consistent format, missing numbers or problematic outliers need to be dealt with – these and other steps of “upfront data transformations and normalization” are often manual, require human judgment, and can take hours of thoughtful attention.  Moreover, the data doctor ought to keep track of all the steps in the clean up.

Modest involvement with “AI software” according to ILTA survey

Signs are everywhere that the U.S. legal industry has started to recognize the potential for computer-assisted decision-making.  For example, the 2016 ILTA/InsideLegal Technology Purchasing Survey had a question on the topic: “Is your firm currently evaluating (or already utilizing) artificial intelligence technologies, systems or related strategies?”  The web-based survey was distributed to 1,231 ILTA member law firms of whom 14% responded (172 firms).

Only 13% of the respondents answered the AI question favorably, consisting of 2% already utilizing such technologies and 11% “currently evaluating” it. Write-ins cited by them include IBM Watson, Kira Systems, RAVN, Lex Machina and ROSS.  Not surprisingly, “half of the respondents that are currently evaluating AI come from Large Firms”, defined as firms with more than 200 lawyers [They comprised 19% of the total respondents.].

What makes it impossible to assess the actual level of support for AI-software is that “Response percentages are based on total responses per question, not overall survey participation” [emphasis added].  Therefore, we cannot say that 13% of 172 firms responded favorably because the survey report does not state how many firms provided an answer to that particular question.