AI Faked Citations In More Than 2,000 Cases As Google Pitches Legal Model

Artificial intelligence continues to produce fake court citations as companies market their language models to law firms.

Generative AI hallucinations were listed in more than 2,000 court decisions since 2023, according to a public database created by legal researcher Damien Charlotin. At the same time, Google just promoted its legal AI tool as a “complement” to legal teams, according to an Axios interview released Tuesday.

“I view it as a complement, not a replacement,” Halimah DeLaine Prado, Google’s general counsel, told Axios. “The practice of law will always fundamentally rise and fall on the exercise of good judgment.”

DeLaine Prado added that AI already assists Google’s legal team with various tasks, such as contract redlining, e-discovery and litigation preparation, and she said these tools could make it easier for small law firms to compete with larger firms, according to Axios.

A spokesperson for Google’s Gemini Enterprise for Legal declined to comment before the time of publication.

“Last year saw, at first, an exponential rise in cases where people were caught hallucinating, but we have now reached a plateau,” Charlotin, a senior research fellow at HEC Paris, told the Daily Caller News Foundation about the findings of his AI hallucination research.

“[There are] several causes for this, amongst which [are] better tools and higher awareness,” Charlotin added. He explained that both frequency and severity of AI hallucinations have plateaued in the past year.

The data currently consists of 2,022 cases of AI-generated content, such as fake citations, used in legal decisions across the globe from April 2023 through Sept. 5 2026, according to Charlotin’s website. More than 1,300 were U.S. cases.

Google’s AI models, specifically Gemini and Bard, were cited in a total of 10 cases, according to Charlotin’s data. ChatGPT was cited over 100 times.