AI hallucination

AI hallucination is the generation of output that is fluent and plausible but unsupported by the model's sources or by fact, including invented citations, quotations, numbers, names, and descriptions of behavior that does not exist.

How it works

A language model generates the next token from patterns in its training and its context, optimizing for plausibility rather than truth, so a well-formed falsehood and a well-formed fact look the same to it as it produces them. Surveys of the failure mode separate two kinds. Factuality hallucination is output that contradicts the world: a fabricated statistic, a citation to a paper that does not exist, an API method the library never had. Faithfulness hallucination is output that contradicts its own provided source, such as a summary that asserts something the document never said. The shared mechanism is that the model has no internal check that a fluent claim is supported, so nothing in generation flags the gap between sounding right and being right.

Why it matters

Hallucination is the failure mode that makes a probabilistic system unsafe to trust on a single output, because the confident wrong answer is indistinguishable from the confident right one at the moment of reading it. It is the reason a draft from an AI is a draft and not an answer, and the reason any output that matters needs a check the model did not author. The honest framing is that hallucination is reduced, not removed: grounding the model in retrieved sources, allowing it to answer "I do not know," and verifying claims against citations all lower the rate, but none make a generated claim self-certifying. A workflow that ships AI output without an external check has decided to ship its hallucinations alongside its correct answers.

In practice

A research assistant is asked for sources on a niche topic and returns three references with authors, titles, and journal names, all formatted perfectly, and one of them does not exist. Nothing in the response signals which citation is invented, because the fabricated one is as fluent as the real two. The fix is not to ask the model to try harder; it is to check each citation against a source the model did not generate, so the invented reference fails a verification step rather than reaching the reader as fact.

Practical considerations

The mitigations are layered, and each addresses a different slice. Retrieval grounds an answer in supplied passages, which attacks factuality hallucination but not a summary that misreads the passage it was given. Allowing the model to say it does not know lowers the pressure to fabricate when the answer is absent. Asking for direct quotes and citations makes a claim auditable, so a downstream check can confirm the quote exists in the source, and a second model judging faithfulness against that source catches the summary that drifted. The throughline is that none of these is self-certifying: each lowers the rate or makes the failure detectable, and the reliable posture is a verification step outside the model rather than a better prompt inside it.

Defined by Ready Solutions AI