AI Faked Citations In More Than 2,000 Cases As Google Pitches Legal Model

Artificial intelligence continues to produce fake court citations as companies market their language models to law firms.

Generative AI hallucinations were listed in more than 2,000 court decisions since 2023, according to a public database created by legal researcher Damien Charlotin. At the same time, Google just promoted its legal AI tool as a “complement” to legal teams, according to an Axios interview released Tuesday.

“I view it as a complement, not a replacement,” Halimah DeLaine Prado, Google’s general counsel, told Axios. “The practice of law will always fundamentally rise and fall on the exercise of good judgment.”

DeLaine Prado added that AI already assists Google’s legal team with various tasks, such as contract redlining, e-discovery and litigation preparation, and she said these tools could make it easier for small law firms to compete with larger firms, according to Axios.

A spokesperson for Google’s Gemini Enterprise for Legal declined to comment before the time of publication.

“Last year saw, at first, an exponential rise in cases where people were caught hallucinating, but we have now reached a plateau,” Charlotin, a senior research fellow at HEC Paris, told the Daily Caller News Foundation about the findings of his AI hallucination research.

“[There are] several causes for this, amongst which [are] better tools and higher awareness,” Charlotin added. He explained that both frequency and severity of AI hallucinations have plateaued in the past year.

The data currently consists of 2,022 cases of AI-generated content, such as fake citations, used in legal decisions across the globe from April 2023 through Sept. 5 2026, according to Charlotin’s website. More than 1,300 were U.S. cases.

Google’s AI models, specifically Gemini and Bard, were cited in a total of 10 cases, according to Charlotin’s data. ChatGPT was cited over 100 times.

Most cases resulted in a warning, with fewer than 160 cases resulting in disciplinary action, according to the data. Most fines for these incidents were between $1,000 and $5,000.

“[A]ll things otherwise equal, free LLMs [large language models], especially those that can’t search the Internet, hallucinate more as a baseline,” Charlotin told the DCNF. “Unfortunately, that’s also the kind of models self-represented litigants (the majority of cases in the database) tend to use.”

In one of the earliest cases in Charlotin’s data, two lawyers cited fake court cases in their legal brief that were generated by ChatGPT before being sanctioned by a New York federal judge.

“[T]he law firm of Levidow, Levidow & Oberman P.C. … abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT,” the court stated in a June 2023 sanctions order.

Even after the court questioned whether the cases were real, the firm continued to defend them, according to the legal order.

Google launched Gemini Enterprise for Legal in August to perform legal tasks that the company said regular large language models (LLMs) cannot accomplish.

“Gemini Enterprise for Legal includes an open ecosystem that extends its capabilities with specialized [AI] agents like Deloitte Legal Assist, bringing deep legal experience to a platform with centralized governance and control,” Google Cloud said in a Sept. 5 X post promoting Gemini Enterprise.

The legal AI model provides “skills that enhance a firm’s own expertise, connections into the systems where matters actually live, agents that complete work rather than return suggestions, and an open ecosystem to extend all of its — with governance running underneath all four,” according to an Aug. 25 blog post from Thomas Kurian, CEO of Google Cloud.