“Dave, I’ve got a question for you: Is traditional predictive coding effective for identifying privileged documents?”
“The short answer is not very well...I am a big fan of predictive coding for helping identify responsive documents and relevant documents…But when it comes to privilege, it’s a different story.
“Privilege is much more nuanced. It has a lot more to do with roles and relationships, and the algorithms that are used for most traditional predictive coding don’t work very well.”
—Dave Cohen, Reed Smith
So begins an eye-opening recent webinar, A Better Way for Harnessing AI for Accuracy & Productivity, hosted by EDRM and moderated by its CEO and Chief Legal Technologist Mary Mack. The discussion focuses on the use of AI to increase accuracy and productivity in privilege review and details the results of the Reed Smith LLP proof of concept that tested Text IQ against their traditional privilege review methodology.
Mack was joined by David Cohen, Reed Smith partner and Records & E-Discovery Group chair, and Omar Haroun, Text IQ’s co-founder and COO.
“Prior to Text IQ, I had not found a product that can do, on an automated basis, a good job at identifying privileged documents.”
AI Fits Well in the eDiscovery Ecosystem
“We’re beginning to see firms like Reed Smith that have really invested in, and are getting knowledgeable about, how to utilize AI-based software tools. They are taking advantage of what machines are good at — rote, more low-value, high-volume work — to support the attorneys combining the best of both worlds."
Text IQ is an AI company, not an e-discovery company. “Our focus as an artificial intelligence company,” says Haroun, “has been on the technology. We’re not a service provider, and we recognize there’s a whole ecosystem out there of people who do that extremely well. This also means that while Text IQ offers a Relativity® plug-in, Text IQ is review software and platform agnostic.
Relativity Plug & Play
Unsurprisingly, “we do work with Relativity the most frequently,” says Haroun. “For those who are Relativity users it would be very easy to visualize…the inputs that are needed for the software are already in your Relativity database in terms of things like the list of attorneys and the extracted text…and then the results Text IQ delivers back can be overlaid into Relativity.”
While the advanced Deep Learning AI behind automating first-pass priv review with better than 99.9% accuracy may be complex, the workflow to leverage it is very straightforward. In the case of the Reed Smith POC:
- Using the Relativity API, Reed Smith provided Text IQ with the extracted text files.
- Text IQ analyzed the data.
- Text IQ populated the resultant potentially privileged tags, scoring, and reason descriptions.
- Results were returned to Reed Smith for seamless incorporation into the review workflow.
Disclosure and Defensibility
The arguments that plagued the use of TAR — perhaps reaching a crescendo when Judge John M. Facciola ruled that search methodology in producing e-discovery if challenged must be scrutinized under Rule 702 in United States v. O’Keefe and their denouement with Judge Andrew Peck’s (retired) 2011 Da Silva Moore ruling — are not really relevant to Priv Review.
“I see no reason to disclose to your adversary that you’re using the process for privilege…it’s very different than predictive coding, because every document we withheld was after attorney review to confirm that yes, this document was in fact privileged. The other side is not prejudiced in any way.”
The concerns regarding defensibility are also put paid. As Cohen reminds, opposing is not prejudiced. Peck also noted Sedona Principle 6, which states that “Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information has ‘been cited favorably in many court decisions.’” More to the point, says Peck, “the question isn’t is your process…good enough for the court on privilege? You know, what would I care, as the judge?” Of course, “if I were the client, I would care very much.”
But he cautions: “Just make sure to explain what you’re doing, what the technology is doing, and why the result—this is after you’ve done it—is reasonable.” And as Laura Kibbe reminds: make sure you document everything.
“You can use Text IQ with a high degree of accuracy, identify privileged documents,” says Cohen, but “I still always recommend the 502(d) clawback agreement, because there’s always a chance — regardless of what method you’re using — of some privileged documents getting through this is even true with human review. In fact, probably more true with just human review.
The Proof is in the Priv
As privilege review requires senior attorney oversight and the creation of privilege logs, how much value does this really offer?
“I have had the opportunity to actually test that out,” says Cohen. “In one situation we just had such a huge volume of documents that a traditional privilege review with lawyers was not feasible.”
“The old traditional method would be to throw a lot of attorneys at them and let them review the documents, maybe low-cost contract or project attorneys for a first pass review, supervised by more senior, more experienced attorneys.”
“So that’s where we decided to do our proof of concept with Text IQ …we supplied information that we would typically use as part of our screen, including the names of known attorneys so that they can pre-populate the brain that they use.”
What Reed Smith discovered is that if they relied only on the privileged screen, they would have inadvertently produced 50 responsive privileged documents to the other side.
Significantly, they found that more than 99.9% of the privileged documents were actually identified by Text IQ. Only “two true misses in terms of documents that got through Text IQ,” says Cohen, after which Text IQ “adjusted the platform so that it would have caught those documents as well.”
Priv Log Support
Text IQ also prepared draft privileged descriptions for each of the documents withheld as privileged. “And we found those descriptions to be really good,” said Cohen. “The text that was prepared by the Text IQ engine was actually much better than our boilerplate in terms of describing the privileged documents.”
“We knew that we needed to provide a result that was interpretable for a human. And this is a big topic in the machine learning world right now, interpretable AI.
“We ensured that every prediction that the machine made, it was actually able to generate a reason in natural language that a human could understand…to help a lawyer make a faster, more accurate decision for themselves.”
“It reduces editing time and cost,” Cohen says. “We use our high-level people to do the privilege, like editing, and to do privilege QC. And so that does add up to significant cost savings.”
“Having the Text IQ generated priv log reduces editing time and cost,” relates Cohen. “We now use our high-level people to do just the priv log editing and review, which adds up to significant cost savings.”
Haroun and Cohen also discuss the qualitative and quantitative benefits of this AI-powered approach highlighting the significant differences between predictive coding (supervised machine learning) and advanced artificial intelligence (unsupervised machine learning) that leverages not just Natural Language Processing, multilayered neural networks, and most importantly, Text IQ’s ability to analyze social context — the key ingredient needed to master the nuance of privilege review.
For more information on Text IQ's AI for Legal, please visit here.