AI Hallucinations in Business: What's Actually Happened So Far

A lawyer submits a legal brief built on six court cases that turn out not to exist. A car dealership's chatbot agrees to sell a $76,000 SUV for a dollar and insists the deal is legally binding. A tribunal orders an airline to pay a grieving customer because its own chatbot invented a discount policy that was never real. These are real cases, and they show how AI hallucinations can create legal and financial problems for businesses.

What Is an AI Hallucination?

An AI hallucination is when a model produces information that sounds correct but isn't, delivered in the same confident tone it would use for something true. Ask a large language model for a legal precedent, a refund policy, or a product spec, and every so often it hands you an answer that's fluent, specific, and entirely made up.

This happens because generative AI doesn't "know" facts the way a database does. It predicts the most statistically likely next word based on patterns in its training data, which is a different task from checking whether something is true. Most of the time, that produces something accurate. Occasionally it produces something merely plausible instead, and the model has no way to flag which one just happened, so from the outside a hallucination looks identical to a correct answer until someone has already acted on it.

Why This Stopped Being a Novelty Problem

Two years ago, an AI hallucination was mostly a funny screenshot. Today it's a line item in a compliance review, because adoption has moved from experimental to genuinely operational. Most large surveys of business leaders now put regular use of generative AI at well over two-thirds of organizations, up sharply from just a couple of years ago.

That shift matters because hallucinations scale with usage. A chatbot answering ten questions a day is a curiosity; one answering ten thousand is a liability surface, where the same low error rate turns into a steady stream of wrong answers that somebody eventually acts on. Once AI moves from a pilot to something customers or regulators interact with directly, an odd wrong answer stops being a one-off and becomes a pattern someone can point to in a complaint. Closing this gap requires active safeguards, which is why tools like Plum AI Secure serve as a critical safety net, catching hallucinations before they turn into financial or legal fallout.

Where AI Has Already Gone Wrong

AI hallucinations have already created high-profile liabilities in legal research and customer service, while broader AI failures are creating financial, legal, and operational exposure across HR, software, healthcare, and finance.

Legal Services & Research: In Mata v. Avianca, a law firm incurred $5,000 in court sanctions after submitting a brief containing six fictional court cases generated by ChatGPT. When asked to verify the citations, the bot hallucinated again to validate them, establishing that unverified synthetic output creates direct legal liability.

Recruitment & HR: Automated hiring tools face sharp regulatory scrutiny from agencies like the EEOC over algorithmic discrimination. While bias is distinct from factual hallucination, automated screeners that inadvertently penalize protected groups create severe corporate liability.

Software Development: Automated coding tools introduce production vulnerabilities when models insert security flaws, logic errors, or unvetted open-source code. Businesses—not the AI vendor—remain fully accountable for any resulting breach damages.

Healthcare & Diagnostics: Medical AI liability centers on diagnostic and clinical support tools misinterpreting patient data. If a clinical assistant hallucinates a treatment protocol, the healthcare provider bears full malpractice responsibility.

Financial Services: Exposure around automated financial guidance is mounting across underwriting and investment advising. Unchecked hallucinations in financial modeling, loan approval systems, or robo-advisors invite regulatory enforcement and consumer class actions.

While hallucinations represent fabricated facts, broader operational risks extend to bias, flawed software, and unreviewed advice. Across every sector, ultimate liability settles on the business that fails to review machine output before relying on it.

When Customer-Facing AI Fails: Chatbots and Defamation

In Moffatt v. Air Canada, a tribunal ordered the airline to pay damages after its chatbot invented a bereavement fare policy that didn't exist, rejecting the airline's argument that the bot was somehow separate from the company. The message was simple: a business owns what its chatbot tells customers.

A separate defamation suit against OpenAI, over a chatbot-invented accusation against a radio host, actually went the other way: the court found no proof of malice or real damages and credited OpenAI's own warnings about accuracy, then dismissed the case. Together, the two show that liability isn't automatic in either direction. It depends on what you warned people about, what customers relied on, and what it actually cost them.

What This Means for Your Business

None of this means AI is too risky to use. It means AI is now sufficiently operational to warrant the same seriousness you'd apply to a supply chain or a payment system. The businesses in these cases weren't reckless outliers; they were doing what most companies are doing right now: using AI to move faster in hiring, service, research, and coding.

Knowing where a hallucination could actually hurt you, putting a human in the loop where the stakes are highest, and documenting what your AI is and isn't authorized to promise won't stop every claim, but it's often the difference between a bad week and a court case.

If you're curious about what your business is actually exposed to when AI is part of how you operate, that's what Plum AI Secure is for. Take a look.

FAQ

What is hallucination in AI?
An AI hallucination is when an AI system gives an answer that sounds plausible but contains false or made-up information.

Who is liable for AI mistakes?
Usually, the business using the AI cannot assume the AI provider will take responsibility for the outcome. Liability depends on the circumstances, including what the system was used for, what the business promised customers, and whether reasonable human oversight was in place.

Can a business be sued for AI mistakes?
Yes. Businesses can face claims when an AI system gives false information, causes financial harm, makes discriminatory decisions, or otherwise produces an actionable result. The legal position depends on the facts and the type of AI error involved.

Does E&O cover AI hallucinations?
It depends on the policy. Errors and omissions insurance may respond to certain claims arising from professional services, but businesses should not assume that AI-related losses or claims are automatically covered. AI-specific exclusions, definitions, and policy wording matter.