The AI Security Review Benchmark: 12 scenarios with graded answer keys.

Drel maintains a 12-scenario capability-evaluation corpus used to test whether its own review engine can produce a sharp, specific, non-generic threat model and clearance decision on a system it has never seen. Each scenario is a realistic-but-fictional system description — deliberately “uncomfortable in the right places”: real autonomy ambiguity, real data sensitivity, real tool blast radius, not a toy “a chatbot answers questions” example. Each has an answer key — the threats a competent reviewer would be embarrassed to miss, the controls a defensible clearance requires, the correct clearance decision, and a set of deliberate false-positive traps — written by an LLM judge and then corrected and signed off line-by-line by a human reviewer.

Each answer key started as a draft and stayed marked provisional until a human reviewer at Drel read the scenario cold, as an analyst would, and wrote down four things: the threats it would be embarrassing to miss, the controls a defensible decision must require, the correct decision, and the generic threats that would be wrong to raise for that specific system (the traps). The draft was then corrected line by line against those notes and only then marked as graded. Scores against a provisional key are reported as provisional.

The engine is scored on what it actually writes, not on matching our internal identifiers: it never sees the answer key or its ids. A key item counts as found when the engine's output contains every concept the item requires, each in any of its accepted phrasings. For example, “prompt injection via retrieved documents” needs both an injection term and a retrieval term. The sets are written to be specific enough that boilerplate does not satisfy them and loose enough that a correct threat phrased differently still counts.

Of the 12 scenarios, only one — the internal knowledge assistant, deliberately scoped as a well-behaved control case to test false-positive discipline — clears for unrestricted production use. The other 11 are on hold, declined, limited to a restricted pilot, or need explicit pre-production conditions before they can go ahead.

This is not evidence that most real AI systems fail review — the benchmark was built to be uncomfortable, not representative. What it does show is that once a scenario includes the access, autonomy, and data sensitivity that real agentic and LLM systems actually have, a defensible review rarely produces a clean, unconditional pass. A review methodology that clears everything it looks at is not being lenient — it is not looking hard enough.

Across the 12 scenarios, the answer keys specify 68 must-appear threats. 84% of them (57 of 68) are rated critical or high severity.

The pattern holds because the scenarios are written around what a system can actually do — a service account with no per-tenant scoping, a tool manifest with write access nobody scoped down, a connected-app token embedded in client-side code — not around generic categories like “the model might hallucinate.” Specific access produces specific, severe findings. Generic categories produce generic, low-severity ones.

A small baseline set of references applies to every scenario in the benchmark, because every scenario is an LLM-touching system: OWASP LLM01:2026, NIST AI RMF, ISO/IEC 42001, and AIUC-1. Beyond that baseline, the applicable framework set is entirely a function of what the system does and what data it touches — not a fixed checklist:

• Healthcare clinical-decision-support: HIPAA, HITECH, FDA
• Autonomous procurement/finance agent: SOX, COSO
• Code/DevOps agent with CI access: SLSA (software supply chain)
• Systems touching payment data: PCI DSS
• Vendor-embedded AI in a regulated fintech context: DORA, PSD2
• Any system handling EU or California personal data: GDPR, CCPA

A review that applies the same fixed framework list to every system — the pattern most self-service checklist tools default to — will either miss the sector-specific obligation entirely, or apply an irrelevant one and waste the reviewer's time defending against a finding that never applied.

The 12 answer keys also specify 37 explicit “false-positive traps” — generic-sounding threats that would be wrong to raise for that specific system, and that a naive, keyword-matching review would flag anyway. The internal knowledge assistant scenario exists specifically to test this: a curated, read-only, no-tool-access assistant should not be flagged for agentic-blast-radius threats that require tool access it does not have.

A review methodology graded only on what it correctly flags will over-flag by default — every generic AI risk category technically “could” apply to any system. Grading against explicit traps is what forces a methodology to say no to a plausible-sounding finding that doesn't actually fit the system in front of it.