Insights From The Inevitable: Trust, But Verify: Validations For Optimizing Workflows With Next-Gen AI For Privilege & PII

Insights From The Inevitable: Trust, But Verify: Validations For Optimizing Workflows With Next-Gen AI For Privilege & PII

"What I think we're mostly talking about here, which is discovery and its variance under the Federal Rules of Civil Procedure, and most of the States have very similar rules. You do not have to have perfection. That would be ideal, but the standard is reasonableness and proportionality."

Honorable Judge Andrew Peck, retired

Technological innovations introduced to improve e-discovery processes have always faced the hurdles of “prove it.” Far from being the old guard luddites of which lawyers are often accused, counsel is faced with proving the efficacy of the technologies and processes to the courts, proving it to opposing counsel (facing what could become an intense and potentially costly resistance), and proving that it is in the best interest of the client (external or internal). 

The use of technology involving artificial intelligence (AI) – and resistance to it – reached an inflection point with Peck’s now famous 2011 Da Silva Moore ruling approving the use of Technology-Assisted Review (TAR). 

As part of The Inevitable series, Peck, now DLA Piper Senior Counsel, sat down for an insightful discussion concerning the use of AI in the realm of privilege review including redaction of PII and sensitive data: a once unthinkable development. (Scanning documents into a repository and using OCR was once unthinkable too as was the now widely used TAR and many other innovations.)

Peck was joined by e-discovery counsel Bobby Malhotra, a member of the  Munger Tolles & Olson LLP litigation practice and Josh Kreamer, AstraZeneca Director of e-discovery. 

Malhotra focuses on complex e-discovery issues, preparedness, planning, and execution, as well as privacy and data security. Kreamer who is focused on e-discovery strategy, is also a faculty member at the e-discovery Institute (EDI) and a member of the New York and DC Bars.

Peck, who served for 23 years as a United States Magistrate Judge for the Southern District of New York is widely regarded for his leadership in eDiscovery from the bench. American Lawyer named him to its list of the Top 50 Innovators of the Last 50 Years as its Judicial E-Discovery Innovator

The webinar Trust, But Verify: Validations for Optimizing Workflows with Next-Gen AI For Privilege & PII can be viewed here.

Still “Reasonableness” and “Proportionality”

More broadly speaking (i.e., in terms of identifying relevant documents), “so long as you've got a good process, one that seems likely to find the material you're working to find if the issue is searching, that is all that the court requires. It helps when you're able to explain the process to the court” and that “you do not need the court’s permission to use a technology," continued Peck.

Peck raises Sedona Principle 6, which states that the ‘Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.’ noting it has “been cited favorably in many court decisions. Just make sure to explain what you're doing, what the technology is doing, and why the result this is after you've done it is reasonable.”

“I often tell lawyers who work with me, ‘don't take your lawyer hat off in a discovery project but at the same time, don't take your common sense hat off either, right?” says Monger Tolles’ Malhotra. And importantly, “you don't need to employ excessively burdensome techniques to validate AI."

Priv Review and PII

"The privilege and PII [personally identifiable information] review process is just fraught with challenges. It's slow, it's tedious, it's expensive.

Privileged calls are very nuanced. You could be dealing with work product protections that could differ from jurisdiction to jurisdiction. You're dealing with agency principles, joint defense, common interest considerations, and not to mention the situations where lawyers are wearing dual business and legal hats…"

—Bobbie Malhotra 

“On the PII side?” continues Bobbie: “a lot of pain points there because inadvertent disclosures of personal or sensitive information can have significant financial, legal, and reputational implications and harms." 

“Historically, a lot of this couldn't be automated, based on my experience, at least with traditional TAR tools. They don't really achieve the heightened level of recall that's required for capturing privileged content… an 80% recall rate may be reasonable and defensible [for relevance review, but] for a privilege review an 80% recall rate is almost never acceptable… The margin of error razor-thin.”

How Do You Develop an AI Defensible Workflow and Prove It?

“That seems to be the $64,000 question,” says Bobbie. “I believe the first step though, ensure that you have the sufficient pre-project safeguards in place: 

  • you have that buttoned-up protective order 
  • you have the right resources deployed 
  • the right set of tools 
  • you understand what your objectives are for that project 

“Second, I think the goal is really to create a reasonable and defensible process with built-in QC and validation steps at each stage. I usually implement some combination of statistical sampling and targeted analysis.”

“Proof is in the pudding, there's just no substitute for running a POC [proof of concept] and seeing the results firsthand, right. Conducting POC using past data sets and just comparing the past human coding with AI-based workflows. 

If you can start to see results where the AI is finding 98, 99% of the documents that humans saw, on a matter, you're starting to feel much better about the process…the proof is really in the data.” 

—Bobbie Malhotra

“And even better when the computer process finds privileged documents that the human reviewers did not find in that prior case,” opines Peck.

And we are able to now do things at scale without spending an arm and a leg,” says Omar Haroun, COO & Founder of Text IQ. “The move to the cloud has also made it much easier to utilize technologies like this. This is no longer something that is just a theoretical possibility. 

You will find the full presentation here and The Inevitable 2020 Series lineup here.