From The Neuron’s posting entitled, “20-Somethings Break AI”:
Anthropic’s Claude AI just pulled a classic AI move: it completely fabricated a legal citation, and then—plot twist—the lawyers actually used it in court. Awkward.
The scene: A Northern California courtroom, where Anthropic’s lawyers are battling music publishers. Claude decides to get creative and generates a citation out of thin air—complete with a made-up title and nonexistent authors. And get this: their “manual citation check” didn’t even catch it.
It’s like the AI equivalent of writing your homework, but then your dog actually eats it—except in this case, the dog actually wrote the homework, and it doesn’t make any sense. Nor does this metaphor, which is increasingly getting away from us…
This is far from the first time AI has fumbled in the legal arena, and it’s not exclusive to Claude. Earlier this week, another judge absolutely roasted law firms for submitting “bogus AI-generated research.” We seemingly see these headlines every week; here’s all the ones Perplexity and OpenAI Deep Research could dig up (remember, these too could have hallucinations in them!).
Anyway, talk about confidence, right? Feels like more of a lawyer problem than an AI problem. Consider this your friendly reminder to always check your work. Twice.