Context-window risks in RAG and how to bound them

In a RAG system, the context window at each request contains:

• The system prompt:trusted instructions set by the system operator, defining the model's role, scope, and behavioural constraints. The system prompt is the highest-trust component.
• Retrieved document chunks: content selected by the retrieval mechanism from the knowledge base, based on semantic similarity to the query. Trust level varies by document source; most implementations do not differentiate trust within this section.
• The user's query: untrusted input from the end user. The query is the lowest-trust component but the most immediate influence on what is retrieved.
• Conversation history (in multi-turn systems): prior turns in the conversation, which may include model responses that were themselves grounded in retrieved content.

The context window assembles these components into a single token sequence that the model processes without inherent structural separation between trust levels. The prompt template provides semantic markers, but the model's handling of those markers is learned behaviour, not a structural guarantee.

The context window is the convergence point for every trust domain in the RAG system. System instructions, retrieved content, and user input meet here. Any weakness in how these domains are separated will manifest as a context-window risk.

Context stuffing is the condition where excessive retrieved content occupies the context window in a way that degrades or overrides the safety and scope instructions in the system prompt. It can be adversarial — an attacker crafts a document whose content fills a large portion of the context window, pushing system instructions toward the edges where models give them less weight — or accidental, arising from retrieval configurations that return too many or too large chunks.

The security mechanism behind context stuffing is the attention distribution effect: LLMs do not give equal attention to all positions in the context window. Content near the beginning and end of the context receives relatively more attention. System prompts placed at the beginning compete with the beginning of the retrieved content; if retrieved content is long, the system prompt's relative influence on the output decreases.

Adversarial context stuffing is a variant of indirect injection. A document crafted to be verbose — containing large amounts of plausible but low-value content — can be designed to fill the context window when retrieved, pushing the system prompt's instructions to a position where they receive less model attention. The follow-on injection payload, placed near the end of the verbose document, then receives elevated attention as a “recency” position.

Data mixing is a context-window risk specific to multi-tenant RAG systems. In a system where multiple users or organisations share a knowledge base and query infrastructure, the context window for any given request may contain retrieved content belonging to different tenants — if the retrieval access control is not enforced strictly at the vector database level.

The data mixing scenario: User A and User B are tenants in a shared RAG system. User B has uploaded sensitive documents to their namespace. User A submits a query that semantically overlaps with User B's document content. If the retrieval mechanism does not enforce namespace isolation at query time — or if the namespace isolation has an edge-case gap — User B's documents may appear in User A's context window and therefore in User A's response.

Data mixing through the context window is particularly difficult to detect because it may not be visible in the response — the model may synthesise from both tenants' content without obviously citing one over the other. The mixing has occurred, but the output does not signal it.

The security review for data mixing requires adversarial cross-tenant retrieval testing: submit queries from Tenant A designed to retrieve Tenant B's specific content, and verify that none of that content appears in the context window assembled for Tenant A's query.

Privilege confusion is the condition where retrieved content appears to the model as having higher authority than it actually has. The model is designed to give the system prompt authoritative weight; retrieved content is supposed to be data. When a retrieved document is formatted to resemble authoritative instructions — a policy statement, a system directive, a role definition — the model may treat it with system-prompt authority.

Privilege confusion is distinct from explicit injection. An injection attack uses imperative language to override instructions. Privilege confusion uses authoritative framing to elevate the perceived trust level of retrieved content without an explicit override command. The effect is similar — retrieved content influences the model's behaviour beyond its intended role as data — but the mechanism is subtler and harder to detect in output validation.

In assessed systems, privilege confusion most commonly arises from knowledge bases that contain actual internal policies — documents that the system was designed to make queryable, which happen to be written in the imperative “you must” / “the system shall” style. These documents are not adversarially crafted, but their format can trigger privilege confusion for certain model and prompt template combinations.

Context window limits — the maximum number of tokens the model can process in a single request — are an accidental control on some context-window risks. When the context window is small relative to the available retrieved content, the retrieval pipeline must select a subset of documents to include. This selection pressure limits how much adversarially crafted content can enter the context window in any single request.

Smaller context windows are not straightforwardly safer — they may cause important documents to be excluded from context — but they do impose a natural upper bound on the context stuffing risk and limit the amount of potentially injected content that can reach the model at once.

The accidental nature of this control matters. Organisations choosing models with larger context windows for quality reasons (better synthesis from more documents) are simultaneously accepting a larger potential context-window attack surface. The security review must note this tradeoff and ensure that other context-boundary controls are scaled accordingly — retrieval budgets, chunk size limits, and output validation coverage.

Context-window risks are addressed by controls at the assembly stage (how the context window is constructed) and the validation stage (what is checked after assembly and after generation).

Assembly controls. Retrieval budget: a hard limit on the number of tokens that can be occupied by retrieved content, leaving a guaranteed minimum space for system instructions. System prompt placement and repetition: place system instructions at the beginning of the context and optionally repeat key constraints at the end (the highest-attention positions). Retrieved content framing: explicit semantic marking that distinguishes retrieved content from instructions at every retrieval block.

Multi-tenant isolation controls. Namespace isolation enforced at the vector database level as the primary control; metadata filter verification as the secondary control. Adversarial cross-tenant testing as part of the pre-deployment review, not just at initial deployment.

Output validation controls. Pattern checks on model outputs for privilege confusion indicators — imperative language, system directive format, role redefinition — that may indicate retrieved content was treated as instructions. Context coverage logging: for each response, log which documents were in the context window, enabling audit of what influenced a given output.

A context-window risk review produces evidence covering the assembly controls, the multi-tenant isolation, and the output validation:

• Prompt template with retrieval budget, system instruction placement, and retrieved content framing documented.
• Retrieval budget configuration — maximum tokens allocated to retrieved content, maximum chunk size, maximum number of chunks.
• Context stuffing test results — adversarial documents crafted to fill the retrieval budget, model outputs evaluated for instruction dilution.
• Data mixing test results (for multi-tenant deployments) — cross-tenant retrieval test queries and verified absence of cross-tenant content in assembled context windows.
• Privilege confusion test results — authoritative-format documents inserted as retrieved content, model outputs evaluated for elevated-trust treatment.
• Output validation configuration — what patterns are checked, what happens when patterns match.

See the Drel RAG security assessment hub for the context-window risk review module with assembly control templates and test cases.