The AI Security Review Benchmark: 12 scenarios, expert-graded.

Drel maintains a 12-scenario capability-evaluation corpus (`evals/corpus/`) used to test whether its own review engine can produce a sharp, specific, non-generic threat model and disposition on a system it has never seen. Each scenario is a realistic-but-fictional system description — deliberately “uncomfortable in the right places”: real autonomy ambiguity, real data sensitivity, real tool blast radius, not a toy “a chatbot answers questions” example. Each has an answer key — the threats a competent reviewer would be embarrassed to miss, the controls a defensible disposition requires, the correct clearance decision, and a set of deliberate false-positive traps — written by an LLM judge and then corrected and signed off line-by-line by a human founder.

Full grading protocol, matching model, and founder sign-off record are public in the repository: evals/corpus/README.md and evals/corpus/FOUNDER_SIGNOFF.md.

Of the 12 scenarios, only one — the internal knowledge assistant, deliberately scoped as a well-behaved control case to test false-positive discipline — clears for unrestricted production use. The other 11 each require a hold, a restricted pilot, or explicit pre-production conditions before they can proceed.

This is not evidence that most real AI systems fail review — the benchmark was built to be uncomfortable, not representative. What it does show is that once a scenario includes the access, autonomy, and data sensitivity that real agentic and LLM systems actually have, a defensible review rarely produces a clean, unconditional pass. A review methodology that clears everything it looks at is not being lenient — it is not looking hard enough.

Across the 12 scenarios, the answer keys specify 68 must-appear threats. 84% of them (57 of 68) are rated critical or high severity.

The pattern holds because the scenarios are written around what a system can actually do — a service account with no per-tenant scoping, a tool manifest with write access nobody scoped down, a connected-app token embedded in client-side code — not around generic categories like “the model might hallucinate.” Specific access produces specific, severe findings. Generic categories produce generic, low-severity ones.

A small baseline set of references applies to every scenario in the benchmark, because every scenario is an LLM-touching system: OWASP LLM01, NIST AI RMF, ISO/IEC 42001, and AIUC-1. Beyond that baseline, the applicable framework set is entirely a function of what the system does and what data it touches — not a fixed checklist:

• Healthcare clinical-decision-support: HIPAA, HITECH, FDA
• Autonomous procurement/finance agent: SOX, COSO
• Code/DevOps agent with CI access: SLSA (software supply chain)
• Systems touching payment data: PCI DSS
• Vendor-embedded AI in a regulated fintech context: DORA, PSD2
• Any system handling EU or California personal data: GDPR, CCPA

A review that applies the same fixed framework list to every system — the pattern most self-service checklist tools default to — will either miss the sector-specific obligation entirely, or apply an irrelevant one and waste the reviewer's time defending against a finding that never applied.

The 12 answer keys also specify 37 explicit “false-positive traps” — generic-sounding threats that would be wrong to raise for that specific system, and that a naive, keyword-matching review would flag anyway. The internal knowledge assistant scenario exists specifically to test this: a curated, read-only, no-tool-access assistant should not be flagged for agentic-blast-radius threats that require tool access it does not have.

A review methodology graded only on what it correctly flags will over-flag by default — every generic AI risk category technically “could” apply to any system. Grading against explicit traps is what forces a methodology to say no to a plausible-sounding finding that doesn't actually fit the system in front of it.