Training data poisoning — OWASP LLM03 explained

It is easy to conflate training data poisoning with retrieval poisoning — both involve malicious content reaching the model through data rather than through the prompt. They are different attack surfaces with different owners and different controls.

Training data poisoning (LLM03, this post) corrupts the data used to train or fine-tune the model itself. The poisoned content becomes part of the model's learned weights — permanent until the model is retrained.

Retrieval poisoning corrupts the knowledge base a RAG system queries at inference time. The model's weights are untouched; the poisoned content is retrieved fresh on each query. See data poisoning in RAG knowledge bases for that surface — the controls do not transfer between the two.

Foundation model training. Almost no enterprise application team trains a foundation model from scratch. The training corpus — typically hundreds of billions of tokens of web-scraped and licensed text — is the model provider's responsibility, and its exact composition is rarely fully disclosed. The practical control here is not technical; it is contractual and evidentiary: the provider's public statements on training data curation, any red-team results specific to poisoning or backdoor behavior, and contractual commitments that survive a vendor security review.

Fine-tuning. Once an organization fine-tunes a foundation model on its own data — support transcripts, internal documents, labeled examples, RLHF-style preference pairs — poisoning becomes a control the organization owns directly. Every example in that dataset contributes to the resulting model's behavior, and a small number of adversarial examples is sufficient to plant a targeted backdoor without measurably affecting overall quality metrics.

This is the surface most reviews under-scope. Teams document that a fine-tune happened and report the resulting eval scores, but rarely produce a record of where each training example came from, who could have contributed to the dataset, and whether the dataset was reviewed for anomalous or adversarial content before training started.

Not all training data poisoning has the same objective, and the objective determines how hard the attack is to find after the fact.

The backdoor case is the one that should worry a security reviewer most. A handful of examples — sometimes fewer than a hundred, in published research on fine-tuning-scale datasets — pairing a rare trigger phrase with a specific output is enough to plant behavior that standard quality evaluation will never surface, because the trigger is rare by design and the eval set almost certainly does not contain it.

Training data poisoning is difficult to catch for a structural reason: the evidence is in the data, not in the model's behavior on the inputs a typical review exercises. A poisoned model passes every standard capability and safety benchmark, because those benchmarks were never designed to search for a specific, attacker-chosen trigger pattern in an otherwise enormous input space.

You cannot red-team your way to confidence against a targeted backdoor you do not know the trigger for. The only reliable control is upstream — reviewing and constraining what went into training — not downstream testing of what comes out.

This is why LLM03 sits awkwardly in application-layer security reviews built around prompt testing and output validation. Those controls are necessary for LLM01 and LLM02 and do essentially nothing for LLM03. The review has to reach further back, into data provenance, before it can produce a defensible answer.

For foundation-model consumers (no fine-tuning):

• Document the provider's public training data and safety documentation as reviewed evidence, not just referenced
• Track contractual commitments on training data practices and any provider security disclosures
• Define a re-assessment trigger for base model version changes, since poisoning risk is re-inherited on every update

For teams that fine-tune:

• Maintain a per-source provenance record for every dataset that contributes to a fine-tune — who supplied it, when, and how it was validated
• Screen fine-tuning data for anomalous or duplicated patterns before training, particularly for datasets with external or crowd-sourced contributors
• Run safety-specific held-out test cases after fine-tuning, not just task-performance metrics — availability poisoning shows up here; targeted backdoors usually do not
• Version and retain fine-tuning datasets so a suspected poisoning incident can be traced back to a specific training run and source
• Restrict who can add or modify entries in a fine-tuning dataset, with the same rigor as a production code change

An AI security review addressing LLM03 for an assessed system should produce:

• Training surface classification — foundation model only, or fine-tuned, and by whom
• Provider evidence — for foundation-model-only systems, the specific documents reviewed and any contractual commitments cited
• Fine-tuning dataset provenance — per-source record, for any system that fine-tunes
• Pre-training validation record — what screening was applied to fine-tuning data before the training run
• Post-training safety evaluation results — held-out safety test case outcomes, distinct from task-performance metrics

This evidence feeds the LLM03 row of the OWASP LLM Top 10 assessment control table and, where a fine-tune is involved, the supply-chain review for the fine-tuning layer.