Dinsmore & Shohl Lifts Contract-Review Recall to 98.2% with Layered AI for Cyber Incident Response

After a client of national law firm Dinsmore & Shohl suffered a cyber incident, its legal team faced an absolute contractual obligation: notify every counterparty whose contract required notice of a security incident, or risk breach-of-contract exposure. That meant reviewing 26,000 contracts for notification clauses whose language was often "tough," "hidden," or "vague," where a typical 80% recall threshold acceptable in ordinary litigation was not defensible. A manual review of that volume was impractical given the time, cost, and risk involved.

Ediscovery Director Peter Pepiton and cybersecurity partner Jennifer Mitchell built a trimodal, sequentially validated workflow on Everlaw: first, search eliminated draft and duplicate contracts, cutting the population from 26,000 to roughly 6,000 final versions; second, EverlawAI Coding Suggestions (a generative AI classifier trained on attorney-drafted criteria and a 120-document ground-truth set) coded the remaining contracts, run three times to improve recall; third, a traditional Predictive Coding model was trained on the AI-validated coding decisions to catch documents with unusual phrasing or buried clauses. Attorneys manually validated precision and recall with sample review after every phase before advancing.

The first Coding Suggestions pass achieved strong recall, and two additional passes improved it further. Adding the Predictive Coding layer and running formal statistical validation raised final recall to 98.2%, a threshold Mitchell deemed defensible for notifying counterparties. Dinsmore estimated a manual review of the contract set would have required roughly 260 attorney hours (over six weeks of full-time work), which the firm avoided, saving the client money and freeing associates and paralegals for higher-value incident-response work.

• Stacking distinct technologies (search, generative AI classification, predictive coding) as sequential filters can push recall well past what any single tool or manual review achieves, especially when near-total recall is legally required rather than negotiated.
• Human-in-the-loop validation after every phase — not just at the end — is what made the higher recall threshold defensible for a high-stakes cyber notification obligation.
• The validated workflow became a reusable, documented playbook Dinsmore can apply to future large-scale, pattern-based reviews such as compliance audits or M&A due diligence.