Getting To New Law: Standardized Quality Metrics

I was at a gathering a while back that happened to include the General Counsel of a Fortune 100 company.  I asked him if he measured ROI on his legal spend.  “No,” he said, “I can’t.”  Why not?  “I can’t measure quality.”

I suspect, though, that he meant “I can’t measure quality yet.”  At various conference panels, I’ve heard more than a handful of the largest firms say that they’re revamping the way they handle their legal spend to be more in line with other cost centers.  This includes standard business metrics related to performance, efficiency, and value.  ROI means measuring return, and return requires estimating value. Value is defined as quality/cost. Therefore, measuring quality is key in the modernization of legal departments, as well as their legal services vendors and/or law firms.

In product valuation, we’re used to all kinds of quality metrics and standardized benchmarks.  As described by Clayton Christensen in The Innovator’s Dilemma and his related writings, market maturity is measured in part by a move from all-encompassing integrated solutions to componentized, modular ones.  The ability to plug in components facilitates finer grain competition via interface standards.  Benchmarks are important because they allow an apples-to-apples comparison.  When different quality metrics are used by different suppliers, there is simply no straightforward way to compare components.

With standardized quality metrics in place, components can be used based on a value curve allowing someone to piece together an appropriate level of quality within a given cost framework.  This is why, for example, we see benchmarks for computer components, such as CPU FLoating-point Operations Per Second (FLOPS), or Mean Time To Failure (MTTF) for memory or hard drives.  The competition at the component level allows us to buy commodity products of varying capabilities and price points while the quality benchmarks help guarantee that the overall system will function correctly.

Several MIT Sloan Management Review articles over the years have looked at quality within the technology realm.  I reference them with an eye toward identifying whether there is anything inherent in technology that might preclude the same approach to measuring the quality of legal products and services.  For example, in 2005, one article, What Quality Means Today, discussed the role of quality in an Internet marketplace; the description seems quite applicable to legal services today:

Today’s highly competitive worldwide marketplace and advances in information technology have created greater customer demand for quality than ever before. The Internet, for example, has made the quality of a company’s products and services transparent to customers. As a result, new offerings not perceived as being of high quality are increasingly likely to fail or become quickly commoditized. What’s more, consumers now view quality as a fundamental measure of their total perception of the product or service as well as of the company, delivery and maintenance network that provides and supports it — a kind of unified “quality-value” metric.

Another Sloan article (2011), How Quality Drives the Rise and Fall of High-Tech Products, describes how quality trumps having an established user base:

In our study, we defined quality as a composite of the brand attributes (such as reliability, performance and convenience) that customers valued. We then derived our measure of quality from the reviews in four of the most respected and widely circulated computer magazines of the time. We selected the personal computer products and services markets because these markets are supposed to exhibit strong network effects. Yet we found that not only did quality prevail in these markets, but network effects enhanced the role of quality. In other words, network effects drove customers to quality and superior brands.

Can legal services possibly avoid these types of metrics?  So far, perhaps — hence the ROI dilemma of the GC mentioned above.  However,  given the market changes happening in legal services and the shift from a producer-driven market to a consumer-driven market, it’s up to corporate counsel to push the field forward. Standardized quality metrics are front and center.

What is the current state of the art?

One of the best-known resources for in-house counsel is the Association of Corporate Counsel.  In 2011, they published a 50+ page guide to value-based management of outside counsel, as well as many follow-on guides.  There and elsewhere, some common performance metrics in legal services include the accuracy of cost and time estimates, “efficiency”, “responsiveness”, etc.  Increasingly, we also see a “big data” approach to metrics from companies, such as Lex Machina and Ravel Law — trying to predict outcome probabilities using data aggregated from many prior cases.

Final outcome prediction is a challenge if there are no accompanying uncertainty calculations, or an inability to map prior examples into the case at hand.  That is, are the cases that led to a prediction sufficiently like a current case, or are there some material distinguishing issues in the present case that might render those predictions useless?  Moreover, the level of granularity is too coarse.  It would be helpful to measure quality throughout the entire process.  For example, in terms of transactional work, it’s way too late to know if an estate plan has errors after the testator has died, necessitating complex legal work to address problems that might have easily been resolved if caught earlier.

Outcome-dependent payment adjustments such as “holdback bucket payments” also miss the mark in terms of measuring work product throughout the process, possibly signaling a mid-course correction.  Conversely, a well-written brief that furthers a client’s intent is amenable to valuation, and may be high quality even if the client loses a lawsuit for other reasons.  Metrics based solely on outcome too easily incorporate factors outside a lawyer’s control, and therefore make it difficult to reward work based on its intrinsic value.  Similarly, while a lawyer may want to measure repeat business and customer satisfaction as a quality metric, this is also quite coarse grain, and insufficient to distinguish between objective and subjective measures of quality.

One Sloan article from 2012, The Case for Standard Measures of Patent Quality, discusses the importance of quality standards for patents but doesn’t list a single measurable quantity.  Quality metrics work best when applied industry-wide to avoid apples vs. oranges comparisons.  Where ViewABill, for example, discusses the importance of quality metrics but doesn’t actually list any specific ones, it’s impossible to use their framework to compare legal service providers.  Similarly, in one law firm consultant’s write-up about useful law firm metrics, nothing is given as a specific, measurable, quality metric.

For corporations with a large litigation profile, a risk portfolio is possible and metrics related to outcome goals can be correlated to process milestones.  For example, DuPont came up with several factors involving early case assessment where outcome goals included not only final payout but also other characteristics important to management, such as how much time was taken by managers involved in the litigation.  Factors even included early assessment of potential jury reaction.  As DuPont highlights, measurement is key to improving both internal and external legal work:

Determining what to measure is a delicate balance between having a thorough metrics system and being overwhelmed by too much data. DuPont looks to metrics as an informational tool to measure performance in cost efficiency (the business of law) and quality and results (the practice of law). Our approach is unusual because it is used by case teams and primary law firms (or PLFs: firms with whom DuPont established exclusive relationships after its convergence process), not just by upper management, to continuously improve performance.

Based on focus groups with department management and in-house and outside counsel, four areas of performance were selected: traditional results (transaction costs, recoveries, and pay-outs); compliance with internal initiatives (for example, ensuring that every matter has an early case assessment and a budget); efficiency (case staffing, disbursements, and cycle time); and peer-to-peer (best practices and use of cost-saving technology). Quantitative measurements were set up, followed by a detailed analysis and interpretation of the data to ensure that the metrics set met our objectives.

Metrics are an integral part of the continuous improvement process, helping ensure that objectives are being met and that the organization is succeeding, and, just as importantly, identifying which programs are not working and where changes should be made. The DuPont metrics tell us more than the cost of improvements to a process or product. They tell us where to focus resources and help us discover the best practices that drive quality and results.

Management with metrics has led to a 40 percent reduction in the average cost of a case filed against DuPont because of the alliance between our in-house staff and PLFs to meet targets. In fact, in many cases we have surpassed our targets. One of the stated goals of our convergence program, for example, was to place 75 percent of new matters within the PLF network. Reports indicate that over 90 percent of new matters are being filed with our PLFs. This indicator is vital for several reasons: it demonstrates our commitment to building a long-term relationship with our PLFs; it gauges the success of the convergence and partnering program; and it increases savings to DuPont when work goes to firms we have selected as price/performance leaders in their practice area or region.

However, it seems that what was not done was a more comprehensive machine learning analysis to see what correlations might exist between outcomes and various discrete, measurable aspects of work product (see below).  And while litigation may have somewhat measurable outcomes like final payout, how might that map to transactional work?  A comprehensive, standardized set of metrics would allow us to maintain quality all along a process, apply appropriate components or “unbundled” services along the way, and accurately weigh cost/benefit trade-offs in purchasing legal services.

Moving Forward

At a legal financial metrics forum held recently, I raised the point that quality metrics are financial metrics.  Certainly, in the push to increase efficiency of legal services, quality is part of the equation, explicitly or not.  Otherwise, for example, disputes could all be handled by a simple coin toss, and we could use the monkey approach to writing legal memos.

Efficiency in legal services without quality is useless.  In discussions with relatively new entrants or innovators to legal services such as Riverview, Novus Law, EverLaw, Legal Onramp, or Seyfarth Shaw, quality control, focusing on measurable quantities and “six sigma” style process improvement, is front and center.  But as far as I can see, there is little happening to establish industry-wide quality metric benchmarks.

How does this lack of standardized metrics hurt the field?  As one startup CEO said to me, it can be insufficient to show a prospective client quality data for his company’s service, if a Big Law partner on the client’s board can be dismissive of the meaning of the metrics, arguing, for example, that other factors are more important.  It’s an apples-to-oranges comparison.

Quality metrics come in several flavors.  For example, many companies that work with document review establish a human-to-human quality evaluation built into the system in a way that tracks, aggregates, and automatically analyzes each reviewer.  A single poor performer, for example, is relatively easy to identify.  But systematic problems also show up if several people make the same mistake.  Statistical sampling of reviews, or selectively duplicating reviews, is sufficient to guarantee high quality output without necessitating human oversight of each individual review.  Once high quality is compared to efficiency, of course, one can arrive at high value work.  This is especially important within a non-billable hour framework where performance is measured by work output per hour, rather than by the number of hours billed per employee.

While aggregated human-to-human evaluation is important, it isn’t the only type of quality metric that matters.  The ability to do some type of automated quality control is important also.  Returning to the comparison with an engineering perspective, one IEEE article states that in most (image) quality assessment approaches, “the main objective is to develop measures that are consistent with the subjective evaluation.”  We can start with simple quantities and migrate to more complicated ones.  Thus, the ability to discern simply if various clauses exist in contracts is easy in a framework of, say, an XML encoded document within a known markup standard (e.g. OASIS LegalXML eContracts TC).  More sophisticated document automation systems allow for pre-drafting error minimization as well as post-drafting analysis.

But the real issue is how to automatically measure the quality of the bulk of legal work going forward – the Consumer Reports or UL of law.  As I said, with commodity products, we look for standard benchmarks (e.g. FLOPS, MTTF, etc.).  We also look for some type of certification (e.g. Good Housekeeping).  But for legal work, what should we measure?

For example, what might we expect in a high-quality estate plan?  Are there common elements in an IP license agreement that we could use to measure quality – say checking for common phrases that have shown to be problematic in case law?  What about the number of citations per 1000 words, or the ratio of controlling to persuasive citations, or the number of supreme court cases cited?  What about the level of argument nesting?  Beyond these static factors, what about contextual elements, such as the number of citations to opinions written by the judge hearing the case, or cases that that judge commonly references?   How might we measure the quality of dispute resolution systems?

For each of these systems going forward, as we increase the role of automation in the legal system, building in quality metrics and automated evaluation guarantees that increasing efficiency doesn’t come at the cost of decreasing quality.  This is as important for large corporate clients managing their Big Law providers as it is for middle-class consumers using an automated online document delivery system, not to mention those who simply can’t afford an attorney at all.

Certainly law seems to be moving toward more empirical work, as seen, for example, in the journal and conferences of the Society for Empirical Legal Studies (SELS).  However,  a cursory examination of that work yields little in terms of the quantification and measurement of quality.  Only by applying quantitative techniques, machine learning, and empirical analysis to the development of standardized quality metrics, can we unlock the efficiency gains in law that technology has provided in other industries while maintaining or improving the quality required for removing existing barriers to adoption.

Note:  I am an angel investor in legal technology and invest in some of the companies included in this article.

This article is the full version of the summary posted on Bloomberg’s BNA site.