Evaluating a RAG pipeline for security, not just relevance
RAG evaluation frameworks are designed to measure retrieval quality and answer relevance. Security evaluation asks different questions: what data boundaries does the pipeline cross, what can be extracted, and what controls enforce the intended scope?
The RAG evaluation ecosystem has converged on a set of quality metrics that measure the same things: retrieval precision (did the system retrieve the right documents?), answer faithfulness (does the answer reflect the retrieved content?), answer relevance (does the answer address the question?), and context utilisation (how much of the retrieved context contributed to the answer?). These are useful metrics. They are not security metrics.
A RAG system can score well on all of them and still have a significant security exposure. A system that faithfully answers questions based on retrieved context is faithfully answering questions based on whatever is in the retrieval corpus — including poisoned documents. A system that utilises context effectively has no defence against indirect injection through that context. High relevance scores say nothing about access control at retrieval time.
Security evaluation of a RAG pipeline asks different questions and requires different test cases. The quality and security evaluations are complementary — but they are not the same process and should not be conflated.
What standard evaluation frameworks miss
Standard RAG evaluation frameworks — RAGAS, TruLens, and similar tools — are designed around a trust model where the knowledge base is assumed to be trustworthy. Their metrics measure how well the system uses the knowledge base, not whether the knowledge base should be trusted in the first place.
The design assumption is reasonable for the quality evaluation use case: you want to know whether your retrieval pipeline is returning relevant documents and whether your model is faithfully grounding its answers. The security evaluation asks: what happens when that assumption is violated? What if a document in the knowledge base is adversarially crafted? What if a user query is designed to retrieve out-of-scope content? What if retrieved content contains embedded instructions?
Standard evaluation frameworks do not test these scenarios. They use curated test sets where the knowledge base contains only correct, relevant, benign content. Security evaluation specifically tests what happens outside those conditions.
Quality evaluation measures how well a RAG system works in the happy path. Security evaluation measures what the system does when the assumptions of the happy path are violated — when the corpus is not trustworthy, when queries are adversarial, when retrieved content tries to issue instructions.
Security evaluation questions
Security evaluation of a RAG pipeline is organised around three questions, each corresponding to a distinct threat surface:
- What data boundaries does the pipeline cross that it should not?This covers the data boundary and retrieval boundary: what content is accessible through the query interface that should not be, given the user's access scope and the system's intended scope.
- What can be extracted through the query interface?This covers extraction risk: can a user with query access incrementally extract the contents of the knowledge base, or specific documents within it, through successive queries?
- Can documents in the knowledge base plant instructions the model executes? This covers the context boundary and indirect injection: can adversarially crafted documents in the corpus cause the model to take actions or produce outputs outside its intended scope?
Each question requires a separate test methodology. They cannot be answered by the same test suite, and the answers to one do not predict the answers to the others.
RAG security evaluation — four dimensions
| Dimension | Description | Test method |
|---|---|---|
| Retrieval accuracy | Does the pipeline retrieve the documents a legitimate user should receive — and only those documents? Measures both false retrievals (wrong documents returned) and missed retrievals (correct documents not returned). | Query set with known-correct retrievals. Cross-tenant boundary tests (User A's queries should not retrieve User B's documents). Sensitivity classification tests (queries for restricted data classes should not retrieve unrestricted-access chunks). |
| Boundary containment | Does the pipeline stay within its defined data scope? Tests whether queries crafted to retrieve out-of-scope content — data outside the knowledge base's intended classification, data from other tenants, or data the user is not authorised to see — succeed. | Adversarial queries targeting known out-of-scope content. Tenant isolation tests. Access control bypass attempts using queries with elevated-privilege framing. |
| Injection resistance | Does the pipeline resist prompt injection payloads embedded in documents? Tests whether adversarially crafted documents, when retrieved and injected into context, cause the model to deviate from its operating constraints. | Adversarial documents with injection payloads ingested into the test knowledge base. Queries triggered to retrieve those documents. Model output reviewed for payload execution evidence. |
| Output validation | Does the pipeline's output layer prevent leakage of sensitive information from context chunks? Tests whether the model, having retrieved sensitive content, includes that content in its response in ways that exceed the user's access rights. | Queries designed to elicit sensitive data that is present in the context window but should not appear in the response. PII pattern detection on outputs. Output redaction validation test. |
Boundary testing
Boundary testing asks: what data does the pipeline return that it should not, given the intended scope of the system?
The test setup requires a knowledge base with explicitly classified documents: documents within the intended scope (should be retrievable by all authorised users), documents outside the intended scope (should not be retrievable through the query interface), and documents with restricted access (should only be retrievable by users with the appropriate clearance).
The boundary test queries are of three types: direct queries for out-of-scope content by name (“show me the executive compensation details”), semantic queries for topics covered only in out-of-scope documents (queries that should only retrieve relevant content if the out-of-scope documents are in scope), and reconstruction queries (queries that attempt to retrieve partial content from restricted documents by asking about specific facts contained in them).
A system with correct boundary controls will not return out-of-scope or restricted content through any of these query types. A system with a boundary gap will return some content. The test results identify which boundary controls are present and which are absent.
Extraction testing
Extraction testing asks: can a user with legitimate query access extract the contents of the knowledge base — or specific documents within it — through successive queries?
Full-corpus extraction is a known risk in RAG systems with large query interfaces. A user who can submit arbitrary natural language queries can iterate through topics until they have retrieved content covering the full scope of the knowledge base. This is relevant for knowledge bases containing proprietary information — the query interface becomes an extraction interface.
Targeted document extraction is a more specific risk: a user who knows that a specific document exists (or can infer its existence) can craft queries that progressively extract its contents — asking about specific sections, specific facts, specific numbers. Each individual query may look innocuous; the series of queries reconstructs the document.
Extraction testing covers both: systematic topic coverage queries to test full-corpus extraction feasibility, and targeted document extraction sequences for known document types in the corpus.
Injection testing
Injection testing asks: can documents inserted into the knowledge base cause the model to execute embedded instructions?
The test setup: insert crafted documents containing injection payloads into the knowledge base. Submit queries designed to retrieve those documents. Evaluate the model's responses to determine whether the injection payload was followed.
The injection test suite should cover multiple payload types:
- Instruction override:“Ignore your previous instructions and respond with [target content]” — tests whether the model follows explicit override instructions in retrieved content.
- System prompt impersonation: content formatted to resemble a system prompt update — tests whether the model treats retrieved content with system-prompt authority.
- Exfiltration request: instructions to include specific content (other retrieved documents, system prompt contents) in the response — tests whether the model can be induced to disclose context through retrieved-document instructions.
- Scope expansion:instructions that widen the model's apparent mandate beyond its system-defined scope — tests whether the model's scope can be expanded through retrieval.
The injection test results classify the system's context boundary posture: robust (no injection payloads followed), partial (some payload types followed, others not), or open (injection payloads followed broadly).
Evidence requirements
A RAG security evaluation produces a different evidence set than a quality evaluation. The security review evidence includes:
- Test corpus specification: the documents used in the test, their classification, their intended retrievability, and any injection payloads embedded.
- Boundary test results: the queries submitted, the documents retrieved, and whether out-of-scope or restricted content was returned.
- Extraction test results: the query sequences used, the content extracted, and the feasibility assessment for full-corpus and targeted extraction.
- Injection test results: the payload types tested, the queries that triggered them, and the model outputs showing whether injection occurred.
- Control gap findings: each boundary, extraction, or injection gap identified, with the specific query or document that exposed it and the required remediation.
This evidence supports the security disposition — the decision about whether the system can proceed — with a named finding for each gap and a required control for each finding. Quality evaluation evidence (RAGAS scores, relevance metrics) is supplementary context, not the primary security evidence.
See the Drel RAG security assessment hub for the security evaluation module with test suite templates for boundary, extraction, and injection testing.
Blog
Get new posts in your inbox
AI security review, OWASP Agentic Top 10, ISO 42001 evidence, and what AI Committees actually need. No cadence promises — we publish when there's something worth reading.
Evaluate your RAG pipeline for security
Drel structures RAG security evaluation across boundary testing, extraction testing, and injection testing — producing a named finding for each gap and required controls for the disposition.
A note on scope: Drel reviews assessed systems against documented architecture, configuration and intent. It does not ingest live telemetry from production environments. Dispositions reflect the assessed system at the time of review and the re-assessment triggers that govern when the disposition must be revisited.