LLM misinformation as a security risk — OWASP LLM09

A quality framing asks: how often is the model wrong, and can we improve accuracy? A security framing asks a different question: given that the model will sometimes be confidently wrong, what is the worst thing that happens downstream, and does a control catch it before it does?

The distinction matters because the two framings point to different fixes. Quality framing leads to model upgrades, better prompts, few-shot examples — all of which reduce the rate of wrong answers but do not bound the ones that still get through. Security framing leads to controls that bound the consequence regardless of rate: source attribution, confidence signaling, human review gates for high-stakes output, and — for agentic systems — independent verification before the model's output is allowed to trigger an action.

“Hallucination” is often used as a single catch-all term, but the underlying mechanisms differ enough that they need different controls.

Confabulation is the case most people picture — a fabricated citation or API call that sounds exactly like a real one. Source conflation and sycophantic drift are subtler and more common in production: the answer is not invented from nothing, it is a plausible-sounding blend or a drift toward what the conversation implies the user wants, and both are harder for a spot-check reviewer to catch than an obvious fabrication.

The severity of an LLM09 finding depends entirely on what the assessed system does with the output, not on the hallucination rate in isolation. Three patterns account for most of the real incidents we have seen in assessed systems:

• Human-in-the-loop decisions made on unverified claims. A support agent, an analyst, or a clinician reads an LLM-generated summary and acts on it without checking the source, because the summary reads as authoritative and citing back to source material takes longer than trusting the summary.
• Agentic systems acting directly on the output. An agent that retrieves a fact and then uses it as an input to a subsequent tool call — a price, a threshold, an identifier — propagates the error into an action with no human checkpoint in between. This is where LLM09 overlaps with excessive agency: the harm is proportional to what the agent is allowed to do with a wrong fact.
• External-facing content presented as authoritative.Customer-facing answers, generated reports, or published content that states a wrong fact with the same confident tone as a correct one, with no visible indication of the model's actual certainty.

There is no single control that closes LLM09 for every deployment — the right mix depends on how the output is consumed.

• Source attribution. When the model cites a document or fact, the citation should be a verifiable link back to the source, not a generated-looking reference. A citation a reader cannot check is not a control, it is a stylistic imitation of one.
• Confidence signaling. Where the underlying model or retrieval layer can expose a calibrated confidence signal, surface it — especially for outputs that read as fact but were produced without a grounding source.
• Human review gates for high-stakes output. Define which output categories require human verification before use, based on the cost of being wrong, not on how often the model is wrong.
• Independent verification before agentic action. For agents that act on retrieved or generated facts, add a verification step — a second lookup, a schema check, a bounds check — before the fact is allowed to drive a tool call.
• Documented limitations in the interface. Users interacting with the system should know what it is and is not verified against, rather than inferring reliability from the model's fluent tone.

Retrieval grounding (RAG) reduces confabulation risk but does not eliminate it — a model can still misread, conflate, or overstate what a genuinely correct retrieved document actually says. Grounding narrows the problem; it does not close the review item.

An AI security review addressing LLM09 for an assessed system should produce:

• Consumption mapping — for each output surface, who or what acts on the output, and what happens if it is wrong
• Attribution design — documented approach to source citation per output surface, and whether citations are independently verifiable
• Review gate definitions — which output categories require human verification before use, and the rationale for the threshold chosen
• Agentic verification controls — for any agent that acts on generated or retrieved facts, the independent check applied before the action executes
• Documented residual risk — the AI Committee's acceptance of the cases where no practical control fully closes the risk, with rationale

For the broader output-handling picture — including LLM01 and LLM02 controls that overlap with this one at the interface layer — see LLM output validation, the controls that actually work. This evidence feeds the LLM09 row of the OWASP LLM Top 10 assessment control table.