A lawyer once filed a court document citing cases that didn't exist, because an AI tool had invented them and sounded completely sure of itself. The story made headlines, and it keeps being repeated for a reason: it's exactly what these tools do when they're used carelessly.
Researchers call it hallucination. It's the most common reason people stop trusting AI at work, and it's worth understanding rather than just fearing.
Why it happens
A language model doesn't look facts up. It predicts what words are likely to come next, based on patterns it learned from enormous amounts of text. Most of the time, likely and true line up. Sometimes they don't, and the model has no built-in alarm for the difference.
So when it doesn't know something, it doesn't say "I don't know." It writes the most plausible-sounding answer, in the same confident tone it uses for things it gets right. That tone is what makes it dangerous.
When it gets worse
Some situations make wrong answers far more likely:
- Obscure or very recent topics. If the model saw little about it, or nothing, it fills the gaps.
- Specific details. Names, numbers, dates, quotes, case numbers, web addresses and academic references are classic trouble spots.
- Your private information. It has never seen your price list or your policies, so it guesses.
- Leading questions. Ask "Why is X the best option?" and it will happily explain, whether or not X is best.
- Long, complicated chats. Instructions get lost and details drift.
What actually helps
You can't get rid of hallucinations entirely. But you can cut them a lot, and make the rest easy to spot.
1. Give it the facts. The biggest single improvement is not asking it to remember, but handing it the source material. Paste in the document, or use a setup that retrieves your own content first, as we describe in our piece on RAG. A model summarising a text in front of it is far more reliable than one answering from memory.
2. Narrow the job. "Answer any customer question" invites invention. "Answer questions about delivery times using only the text below, and say you don't know if it isn't there" is much safer. Smaller, clearer tasks give better results.
3. Make it admit uncertainty. Tell it explicitly that "I don't know" is an acceptable answer. Models often try to please, and permission to decline reduces made-up replies noticeably, though not perfectly.
4. Ask for sources, then check them. Have it quote the passage it used, or say where in your documents the answer came from. This makes errors visible. Be warned that models can invent references too, so confirm that a cited source exists and really says that.
5. Put a person in the right place. You don't need a human to read everything. You need one on the output that carries risk: legal wording, medical or financial information, prices, promises to customers. Low-stakes drafts can go through with lighter checks.
Other things worth trying
Ask the same question twice, or in a slightly different way, and compare. If the answers disagree on a fact, treat it with suspicion. Lower the "temperature" setting if your tool exposes it, since that makes output less random. And for calculations, let the model call a calculator or run code instead of doing arithmetic itself, because pure text prediction is shaky with maths.
What to tell your team
Treat AI output like a confident intern's first draft: useful, fast and sometimes wrong. Don't paste a figure, quote or reference into something important without checking. If something sounds too neat, check it twice. And report errors, since a shared list of "things it got wrong" helps everyone calibrate how much to trust it.
Know the limits
Some work isn't suited to current tools at all. If a mistake could harm someone, such as medical advice, legal filings or safety-critical decisions, the model can assist an expert but shouldn't replace one.
