LLM supply-chain risk — models, weights, and dependencies

Traditional software supply chains — code dependencies, build tools, base container images — are well understood and have mature tooling: SCA scanners, SBOM generation, dependency pinning, vulnerability feeds. The AI supply chain shares some of these properties and adds several that traditional tooling does not cover.

The additions: pre-trained model weights are opaque — you cannot read them the way you can read source code. Training data provenance cannot be inferred from the model artifact. Alignment and capability claims are made by the model provider and cannot be independently verified from the weights alone. The inference provider processes your prompts and completions — its data retention practices determine whether your data stays within your control boundary.

A standard software SBOM tells you what code is in your application. An AI supply chain review must also document what the model was trained on, who processed the inference, and what capabilities the plugins add — none of which appears in a traditional dependency manifest.

The five layers of the AI supply chain require different review questions and produce different evidence. The sections below address each in turn.

The base model is the pre-trained foundation the application is built on. For most organisations this is a third-party model from a major AI provider — GPT-4, Claude, Llama, Gemini, or equivalent. The risks at this layer are not exploitable through the application code; they are properties of the model itself that the application must account for.

Training data provenance.What data was the model trained on? The provider's documentation may describe the general composition of the training corpus but rarely provides a complete enumeration. For most foundation models, the training corpus includes significant quantities of web-scraped content with unknown provenance. The implication: the model may have memorised content that appears in the training corpus, including PII that appeared in public web content. For most deployments this risk is manageable. For deployments handling regulated data or operating under strict copyright constraints, it is material.

Alignment and safety claims.Model providers publish safety evaluation results, red-team findings, and alignment documentation. These are self-assessments — they represent the provider's claims about the model's behaviour, not independent verification. A security review should document that these materials have been reviewed and assessed against the deployment context, not just acknowledged.

Capability claims.The model may have capabilities beyond what the deployment requires. A coding assistant built on a general-purpose model with medical knowledge, legal reasoning, and synthesis capabilities has excess capabilities that could be exploited if the model is manipulated. The base model's capability profile should be compared to the deployment scope.

Fine-tuning adapts a base model to a specific task or domain using additional training data. It adds two distinct risk surfaces: the fine-tuning dataset itself (which becomes part of the model's effective training corpus) and the alignment risk from fine-tuning altering the model's safety-relevant behaviour.

Fine-tuning dataset provenance.Every example in the fine-tuning dataset contributes to the model's learned behaviour. Poisoned examples — crafted to teach the model specific behaviours on specific trigger patterns — are indistinguishable from legitimate examples in the dataset unless the dataset is reviewed. The review question is: what is the provenance of each data source in the fine-tuning set, and was each source reviewed for anomalous content?

Alignment drift from fine-tuning. Fine-tuning on a narrow task corpus can weaken alignment properties present in the base model. A model fine-tuned to be maximally helpful at a customer service task may follow user instructions more aggressively than the base model, including instructions that cross safety boundaries. Post-fine-tuning behavioural evaluation should include safety-relevant test cases, not just task-performance metrics.

Training data as sensitive data. Fine-tuning corpora built from internal documents may contain sensitive information that the resulting model can reproduce. See the sensitive information disclosure article for the memorisation risk and controls.

Every prompt sent to a hosted inference provider and every completion returned is processed by the provider's infrastructure. The inference provider sees the full content of every interaction — including system prompts, user messages, and retrieved documents that are injected into the context. This has direct implications for data handling obligations.

Data retention.Does the provider retain prompt and completion data? For how long? Under what conditions? Providers vary significantly — some retain data for abuse monitoring, model improvement, or commercial purposes unless the customer opts out or purchases an enterprise agreement that includes a zero-retention SLA. The retention terms must be reviewed against the organisation's data handling obligations, particularly for any prompts that may contain personal data.

Subprocessors.Inference providers use subprocessors for infrastructure, logging, and operations. The subprocessor chain affects the geographic scope of data processing, which has implications for GDPR adequacy decisions and other jurisdictional requirements. The provider's subprocessor list should be documented and assessed.

Terms of service alignment. API terms of service impose constraints on how the model can be used — prohibited use cases, output restrictions, ownership of fine-tuned derivatives. These constraints must be evaluated against the deployment use case before production deployment.

Plugins and extensions extend the model's capabilities through tool calls. Each plugin is a dependency with its own supply chain: the plugin code, the external services it connects to, and the permissions it exercises. The tool manifest — the set of plugins available to the model — defines the model's effective capability scope.

Third-party plugins present specific risks. A plugin sourced from an external developer may change behaviour on update, may connect to external services that change their behaviour, or may contain intentional backdoors. The tool descriptions in a plugin manifest influence how the model decides when to invoke the plugin — a manipulated tool description can cause the model to invoke a tool in unintended contexts.

Review questions for each plugin: What is the plugin's source and update mechanism? What permissions does it exercise? What external services does it connect to? What data does it process? Is the tool description accurate and appropriately scoped?

The software dependencies that connect the application to the model — LLM SDKs, prompt engineering libraries, vector database clients, embedding libraries — are the layer most similar to traditional software supply chain risks and the layer most amenable to existing tooling.

Standard SCA scanning covers known CVEs in these packages. Beyond CVEs, the AI dependency surface has some additional considerations: prompt injection libraries that promise injection detection may have bypass rates that are not publicly documented; embedding libraries may process input data in ways that are not transparent; model fine-tuning frameworks may load weights from URLs specified in configuration files — a configuration injection vector.

A supply chain review for an LLM application must produce evidence across all five layers. The checklist below is the minimum required for each.

The OWASP LLM Top 10 Assessment provides a structured framework for collecting this evidence and incorporating it into the clearance decision for the assessed system.