AI security review vs penetration testing — different questions

The core distinction is this:

• A penetration test asks: can this system be exploited? It tests whether known attack techniques succeed against the deployed system. Success means finding a path from attacker to impact.
• An AI security review asks: should this system go to production, and under what conditions? It evaluates whether the system has been designed, configured, and bounded in a way that makes it appropriate to operate at the stated risk tolerance.

These questions are related. A pentest may reveal that controls assumed to be present in the review are absent or bypassable. A review may define the attack surfaces the pentest should prioritise. But they are not interchangeable, and running one does not make the other redundant.

A penetration test that finds no exploits does not mean the system should be deployed. It means the tested attack paths did not succeed on the day they were tested. The clearance decision requires more than that.

A traditional penetration test of an AI application typically covers:

• Infrastructure and API security. Authentication, authorisation, rate limiting, input validation at the API boundary, data leakage through HTTP responses.
• Model interface exploitability. Direct prompt injection attempts, jailbreak techniques against the production system, system prompt extraction attempts.
• Output handling vulnerabilities. Whether model outputs are rendered unsanitised in contexts where they can cause client-side harm (XSS via LLM output, for example).
• Tool call security. For agentic systems, whether tool invocations can be hijacked or elevated beyond intended permissions.

LLM-specific penetration testing has matured significantly. Frameworks like OWASP LLM Top 10 and the OWASP Agentic Top 10 give testers structured attack catalogs. A good LLM pentest is meaningfully different from a traditional web application pentest — it requires testers who understand prompt engineering, model behaviour, and the specific attack surfaces of the model type in use.

An AI security review is a design-time assessment. It operates on the architecture, the configuration, the data flows, the threat model, and the control plan — not on a live instance of the running system. It produces a clearance decision: a documented judgment that the system meets (or does not meet) the threshold to operate in the stated deployment context.

The domains a review covers that a pentest does not:

• Threat model completeness. Does the threat model account for all relevant attack surfaces — including surfaces the pentest will not test, like supply chain, training data, and indirect injection through knowledge bases?
• Control coverage of the threat model. For each identified threat, is there a specified control? Does each control have an owner, a verification method, and a lifecycle gate?
• Residual risk acceptance. For risks that controls do not fully close, is there an explicit, named acceptance by a person with authority to accept it?
• Data handling and privacy review.Does the system’s data flows comply with the applicable data protection framework? Are there control gaps in how personal data is processed by the model?
• Deployment context fit. Is this system appropriate for the user population, the data classification, and the organisational risk tolerance it is being deployed into?
• Re-assessment triggers. What changes to the system or context will require a new review?

The output of a review is not a list of exploits found. It is a disposition memo that captures the clearance decision, the required controls, the accepted residual risks, and the conditions under which the clearance must be revisited. That memo is the artifact a regulator, an auditor, or a post-incident investigator will ask for.

The overlap between a pentest and an AI security review is real and valuable. In practice:

• The review’s threat model informs the pentest scope. A review that identifies indirect prompt injection via retrieved documents as a key risk tells the pentest team where to focus.
• The pentest’s findings feed back into the review. A pentest that succeeds in bypassing a control listed in the disposition as “in place” is evidence that the clearance was issued on a false assumption. The disposition must be updated.
• Both contribute to the evidence pack. The review produces the clearance decision and the control plan. The pentest report is an evidence artefact that supports or challenges the control verification entries.

For a typical AI system going to production, the combined cycle looks like this:

1. Design-time review (weeks 1–3).System boundary is documented. Threat model is produced. Control plan is drafted and assigned. Residual risks are identified and accepted. Clearance decision is issued — often “restricted pilot” at this stage, with a list of controls that must be in place before full production.
2. Pre-production pentest (weeks 4–5). The pentest is scoped against the threat model from the review. The pentest team is given the control plan so they know which controls are claimed to be in place. Findings feed back into the control plan as open items or as verification evidence.
3. Control closure (weeks 5–6). Pentest findings that represent control gaps are remediated. The disposition is updated to reflect the closure status of each required control.
4. Go-live clearance (week 6+).The disposition is updated to “cleared for production” once the required controls are verified as in place. The pentest report is attached to the evidence pack.

This sequence produces a defensible record at each stage. The pentest is not a substitute for the review, and the review does not make the pentest redundant. Each does what the other cannot.

When a governance committee, an auditor, or a regulator asks for the evidence that an AI system was properly assessed, the two disciplines contribute different artefacts:

The two evidence sets are complementary. A governance pack that contains only a pentest report shows that the deployed system was tested for known exploits. It does not show that the system was reviewed against its risk profile, that residual risks were accepted by accountable parties, or that there is a process for re-assessment when the system changes. A pack that contains both shows all of that.

The most defensible posture is a review that sets the design-time baseline, a pentest that verifies the runtime implementation of the controls, and a disposition that records both and ties them to the clearance decision.