Law firm Cozen O’Connor has a rule against using publicly available chatbots to draft legal filings. But after a judge penalized two of its lawyers for citing fake cases, the firm is adding some extra protection: an AI hallucination detector.
Cozen O’Connor is now testing software, made by a startup called Clearbrief, that scans legal briefs for made-up facts and produces a report. Think spell-check, except instead of flagging typos, it spots the fictional cases and citations that generative tools sometimes invent.
“You have to be pragmatic,” said Kristina Bakardjiev, the Cozen O’Connor partner tasked with harnessing technology to serve lawyers and their clients. She said lawyers will play around with chatbots whether the tools are authorized or not.
Stung by embarrassing AI hallucinations, the legal field has adopted bans on general-use chatbots and AI assistants. But it’s hard to stop a curious associate from pasting a draft into a free, browser-based chatbot like ChatGPT, Claude, or Gemini. Now law firms and legal tech companies are scrambling to lower the risk of bogus citations and catch those that sneak through before they land in front of a judge.
Two of Cozen O’Connor’s defense lawyers in September admitted they had filed a document riddled with fake cases after one of them used ChatGPT to draft it, against firm policy. A Nevada district court judge gave the firm a choice: remove the lawyers from the case and pay $2,500 in sanctions each, or have the pair write to their former law school deans and bar authorities explaining the fiasco and offering to speak in seminars on topics like “professional conduct.”
Both lawyers went with option No. 2. Cozen also fired the lawyer who had used ChatGPT.
Earlier this year, Damien Charlotin, a legal data analyst and consultant, began tracking cases in which a court had discovered hallucinated content in a legal filing. Charlotin tallied 120 cases between April 2023 and May 2025. By December, his count had hit 660, with the rate of new cases accelerating to four or five per day.
The number of documented cases remains small relative to the total volume of legal filings, Charlotin said. Most cases in his database involved self-represented litigants or lawyers from small or solo firms. When large firms were involved, the hallucinations often slipped in through the work of junior staff, paralegals, experts, or consultants, or through processes like formatting footnotes, Charlotin said.
Hallucinated content is causing headaches in other professions, too. In October, consulting firm Deloitte agreed to pay a partial refund to the Australian government for a $290,000 report after officials found it was peppered with allegedly AI-generated errors.
Straying from the walled garden
AI hallucinations are hard to eliminate because they’re baked into the way chatbots work. Large language models are trained to predict the word that is most likely to come next, given the words before it.
Michael Dahn, a senior vice president at Thomson Reuters who leads global product teams for legal-research service Westlaw, says the model makers can’t get hallucinations to zero for answering open-ended questions about the world. However, companies can dramatically reduce their risk by forcing a large language model to cite from a specific data set, like a corpus of case law and treatises. The model can still mismatch or overlook content, but wholesale fabrications are far less likely.
Thomson Reuters and LexisNexis are selling that promise to customers: that an artificial assistant confined to their walled gardens of vetted material is safer than a chatbot trained on the open internet. Both companies have spent decades and heaps of money building deep repositories of case law and other legal content. More recently, they’ve bolted on AI-powered tools to help lawyers search and cite their data. They now have to defend their positions against services like ChatGPT and Claude that are creeping into the legal field.
LexisNexis has also extended its moat to Harvey, the legal tech startup whose valuation has climbed to $8 billion. Harvey struck a partnership with LexisNexis this year that pipes one of the world’s biggest legal databases into Harvey’s generative tools.
Harvey also works with AI model providers, such as OpenAI and Anthropic, to constrain which datasets they’re allowed to draw from and layer in Harvey’s own proprietary datasets, a spokesperson said. Lawyers can then inspect logs that show how an answer was reached and what data fed into it.
An AI fact-checker
Clearbrief makes a drafting tool for litigators that works as a Microsoft Word plug-in. Jacqueline Schafer, a former litigator who founded Clearbrief, says its product detects citations using natural language processing, and creates links to the relevant case law or documents from the case. The tool calls out citations and facts that are fabricated or contain typos. The tool also points to places where the underlying source doesn’t quite support what the writer claims.
Cozen O’Connor has been testing a new Clearbrief feature that lets users generate a cite-check report before passing a draft to a partner or filing it in court.
Schafer says partners at large firms trust their junior staff to check citations rather than vetting every case themselves. Still, federal rules hold the partners who sign filings personally responsible for their accuracy.
Part of Clearbrief’s appeal for Cozen O’Connor is the paper trail. The firm is upgrading its knowledge management system, and Bakardjiev imagines that someday the firm might store cite-check reports alongside drafts and final filings, creating a chain of custody for every brief.
If a judge ever asks what a partner did to prevent hallucinated citations, Bakardjiev said, partners can point to a report that shows who ran the check and when.
The legal world is likely to live with hallucinations for a long time. The unglamorous part of the solution is training lawyers to treat the chatbot output as a starting point, not the finished work. The other answer: throwing more AI at the AI.
Have a tip? Contact this reporter via email at mrussell@businessinsider.com or Signal at @MeliaRussell.01. Use a personal email address and a non-work device; here’s our guide to sharing information securely.
Read the full article here
