I was asked to participate on a panel at the ABA’s Center for Professional Responsibility’s 2nd UPL School in Chicago on April 17th. The panel topic was “The Users and Abusers: Technology and the Unauthorized Practice of Law“. I’d written a guest blog at MyCase in 2013 called “Using A Document Automation System – Authorized Practice Of Law?” In it, I’d questioned whether document automation could really be considered “unauthorized”, depending on the particular implementation, if the software was simply instantiating what a lawyer would have produced. The panel gave me an opportunity to update my thinking about it, in front of a compelling audience of state bar executives, ABA ethics experts, and even the GC of LegalZoom.
As might be clear from some of my recent posts and talks, I’m on a quality metrics kick these days. It’s the rare issue that arises as frequently when talking about Big Law, as it does for consumer law, as it does for Access to Justice (A2J). I started measuring quality back in my Computer Science dissertation, and continued when I got to Google and started the search quality evaluation group. That work required me to develop metrics that jibed with our subjective notion of quality. Legal technology is no different – it requires metrics and a evaluative framework.
Compared to What?!
A lot of the discussion around UPL and technology seems to boil down to the assumption that software produces lower quality work product than lawyers do. Depending on the audience, it seems to be outright offensive to even raise the issue. But before we can rationally object to the putative lower quality of technology, we need to ask the question about the quality of lawyer-produced work product. Lawyers and law firms often argue that they produce the highest quality work, as compared with other lawyers. That is, by our own account, we don’t think that all lawyers produce the same quality work. If some work is better, some must be worse. When we object to the quality of technology, with which lawyer are we comparing? The threshold for quality for lawyers is malpractice, based mainly on standards for competence and negligence. As anyone who has studied criminal procedure knows, merely poor work is insufficient for a criminal defendant to receive a new trial. The point is that there is a range of allowable quality of legal work performed by licensed attorneys. Why should technology have to perform better than the worst allowable lawyer?
One complaint that came up in the panel discussion was that judges and lawyers have seen poor results from software – usually related to estate plans or bankruptcy. While that complaint may be valid, does it imply that we should throw out the baby with the bath water? What if software produces at least “good enough” work product in areas such as eviction, uncontested divorces, or debt collection, all better than consumers generally do without the software? Wouldn’t it be better to limit the software by practice area than to claim UPL for all of it? Moreover, I’ve heard similar complaints by lawyers (for example at Wilson Sonsini and Fenwick) about clients coming in with poorly structured incorporation documents, but created by other lawyers. Why would we tolerate poor work product from some lawyers and yet not argue that we should shut them all down, but then turn around and make that argument about software?
Let’s Be Honest
Let’s assume for the moment that we compare technology to the “reasonable” attorney, however determined, and find the technology to have lower quality. Are we done? Does it have to be better than the hypothetical lawyer in order for it to leave clients better off than the real-world alternatives they face? The fact is that the access-to-justice problem has been discussed so broadly, and published so widely (by the ABA, LSC, and on and on), that I’d argue that it’s literally unconscionable to pretend that millions of people are not representing themselves and failing miserably in the process. The issue is not whether technology can compare with a given attorney, or even the worst tolerable one. The issue is whether technology leaves the client better off than viable alternatives. Where the alternative for a particular person is no lawyer due to a lack of personal or societal resources, then the technology needs to be compared with that individual going it alone. No other comparison is rational. And for anyone who wants to compare technology with lawyers, I ask how we measure the quality of lawyers to make the comparison. Having no data isn’t an excuse to block technology that can help people with no other choice.
Advantages of Technology
What are the advantages of software-based legal help? The first, obvious, advantage is how inexpensive it is. Even LegalZoom or DirectLaw fees, for-profit enterprises, are in the hundreds, not thousands of dollars. (Of course the key here is triage to be able to exclude cases that are too complex for the software to handle.) Another benefit is the ability to correct software when problems are found, not to mention that problems are more likely to be discovered with software than with humans since interactions can be logged and analyzed. Also, software can be certified just like people, but, once that’s done, the behavior is quite predictable. Software can also be classified, so that some, like yes/no decision trees, can be accurately predicted and tailored with high certainty to specific cases (e.g. types of eviction, bankruptcy, divorce, incorporation, etc.). Other technology that is more fuzzy might be available only if accompanied with lawyer review, and thus gain in efficiency while bypassing any UPL issues at all. Finally, quality control is at least as available for software as it is for people. Just because we get along with a professional doesn’t mean the work product is good. For software, we’re more likely to measure the work product than simply the interface – the interface is more of a market issue, while the work product is more of a licensing issue.
Keep Your Eye on the Prize
In computer science, we talk about the Turing Test as a way of deciding the level of AI of a system. If you talk with something without seeing who/what it is, and you can’t tell if it’s a person or a computer, it passes the Turing Test. Well, if you look at legal work product and you can’t tell if it was generated by software or a lawyer, really, how bad could the software be? The goal of UPL is to protect clients, not to prevent them from getting help. If we focus on measuring the quality of work product and not the process we use to get it, we stand to help a lot of people who really need it.
When I worked for a legal process outsourcing company (LPO), prospects often said “Your quality won’t be good enough”. I would then ask, “how do you measure your quality now?”. That question was met with blank looks – almost every time.
Lawyers have the reputation of being highly analytic. I’m not sure it’s deserved. If you question quality, you need to be able to answer how you measure it. Or if a lawyer states, for example, “predictive coding won’t work”, he or she needs to understand what the next best alternative is. I’ve rarely encountered lawyers who can articulate or understand the idea that they most compare alternative approaches.
The issue you raise here is conceptually the same as for predictive coding. We don’t have a “gold standard” (used in the medical clinical trial jargon sense) for most of what lawyers do.
Completely agree, Ron. When it comes to quality, there are no metrics, no corpus of data, no evaluation frameworks, etc. The typical response I get when I ask lawyers how they measure quality is that “our clients do not complain”. This is not much of a metric. How is the single mom going through a family law case supposed to evaluate quality? How is the beleaguered in-house counsel (who juggles multiple documents and trusts outside counsel) supposed to evaluate quality? There are many variables at play, so relying on the outcome of court cases is dodgy at best.
In my own admittedly limited experience reviewing the work products of lawyers, I’ve seen some that are a dog’s breakfast, and other documents that are exquisitely crafted.
“The issue is not whether technology can compare with a given attorney, or even the worst tolerable one. The issue is whether technology leaves the client better off than viable alternatives. ”
I would agree with this. Given the scandalous levels of unmet demand for legal services, it is hard to see the response of lawyers towards platforms like RocketLawyer and LegalZoom as anything but protectionism.
If the auto industry was as good at innovating and meeting customer demand as law firms, 5% of today’s population would have Ford Model T cars that required hand cranking and became more expensive to purchase each year, while 95% of the population would be walking on foot.