What evidence can we demand from AI vendors?

At minimum: a data processing agreement that explicitly covers AI use of customer data, a sub-processor list with change notification, and a data retention policy. Better: a SOC 2 Type II report whose scope includes the AI feature, an ISO 27001 certificate, a model card, and a recent penetration test report. Best: an independent assessment of the AI feature against OWASP LLM Top 10 or OWASP Agentic Top 10, and a defined incident disclosure policy.

How does Drel handle vendor SaaS AI?

Drel reviews the vendor AI system based on what the vendor has documented and disclosed. It maps the documented architecture, declared data flows, and contractual control claims to required controls and produces a clearance decision with explicit conditions. Each evidence gap (something the vendor has not documented or evidenced) is recorded as a control gap in the disposition.

What if the vendor refuses to share design artifacts?

Refusal to share documentation is itself an evidence gap and goes into the assessment record. The conditional clearance can include a requirement that the vendor provide specific documentation before moving from pilot to production. A vendor that consistently refuses to provide documentation is producing a control gap that the AI Committee must explicitly accept as residual risk or reject as a blocker.

Does Drel certify vendors?

No. Drel produces a structured assessment of the vendor's system based on available documentation. The output is your record of having assessed the vendor — it documents what you reviewed, what you accepted, and what you required. Certification is a separate, accreditation-based process performed by audit bodies, not by review platforms.

What triggers a re-assessment of a vendor system?

The vendor changes their model or model version, adds training data sources, changes sub-processors, updates their data handling policy in ways that affect your use, has a publicly disclosed AI-related incident, or expands the scope of the AI feature (new data types, new use cases). Each trigger should be named in the original disposition with an owner.

How does the EU AI Act apply to vendor AI?

Under the EU AI Act, deployers (organisations using AI systems) have obligations distinct from providers (who built the system). For high-risk AI systems used in your operations, you retain deployer obligations including monitoring use, ensuring the system is used per provider instructions, and informing affected persons. The vendor AI assessment record supports the evidence for these obligations.

Vendor AI security assessment — without access to the vendor's source.

You cannot audit a vendor's model weights. You can audit their documented architecture, data handling policies, control claims, and evidence of those claims. That is a vendor AI security assessment.

Drel18 May 20269 min read

The vendor AI assessment problem

Procurement is moving faster than security review. A line-of-business team buys a SaaS product that quietly added an AI feature. A new vendor pitches an AI-first workflow. An existing vendor announces that they will be sending customer data to a third-party model. In each case, the security review that was performed at original procurement either did not cover the AI surface or did not exist. The system has changed; the disposition record has not.

Vendor AI assessment is the structured review you can run when you cannot see the source code, cannot inspect the model weights, cannot audit the training data, and cannot access the runtime configuration. What you can review is documented architecture, declared data handling, contractual control claims, and the evidence the vendor will provide that those claims are operating.

The output is the same shape as any other Drel assessment — a threat model specific to the vendor deployment, a control plan, an evidence gaps report, and a clearance decision. The difference is that the controls split into two columns: controls implemented by the vendor (where evidence comes from vendor disclosures) and controls implemented by your organisation (where evidence comes from your own records).

What you can assess from vendor documentation

Without source-level access, six categories of evidence are available, and each one is something you can demand in writing as part of procurement:

Documented architecture.A system description that names the model, the prompt structure, the tools the model has access to, the data flows between user input and the model, and any retrieval or memory layer. “We use AI” is not a system description. A system description names specifics — provider, hosting region, version, and configuration.
Data handling policy. What customer data enters the model context. What is retained, where it is retained, for how long, who can access it. Whether prompts and outputs are used for model training. What happens to data if you terminate the contract.
Contractual control claims. A list of the security controls the vendor commits to maintain. These belong in the contract or in a referenced security addendum, not in marketing pages. A control claim that is not in a contract is not an enforceable control.
Evidence of control claims. SOC 2 report (preferably Type II), ISO 27001 certificate, penetration test summary, recent third-party AI assessment if available. Each document covers different controls; none alone covers AI-specific risk.
Sub-processor list and accountability. Which third parties the vendor relies on to deliver the AI feature. Model providers, hosting infrastructure, observability platforms. A sub-processor list with a date and a notification policy when it changes.
Incident history and disclosure policy. Has the vendor disclosed any AI-related incidents? What is their disclosure policy if an incident affects your data? What is the notification window? What evidence will they provide to support post-incident response?

What evidence to demand and what it covers

Vendors often offer standard documents that imply broad assurance. Each of those documents covers a specific scope, and most do not cover AI-specific concerns. Reading them honestly:

SOC 2 Type II — covers operational controls over availability, security, confidentiality, and (sometimes) privacy. It does not typically cover AI-specific risks like training data provenance, model behavior under adversarial input, prompt injection handling, or retention of model context. Ask whether the SOC 2 scope explicitly includes the AI features you are buying.
ISO 27001 certificate — covers an information security management system. Useful evidence of organisational security maturity. Does not address AI-specific risk management. If the vendor also claims ISO 42001, ask for the certificate and the scope statement.
Penetration test report — covers application-level vulnerabilities. Penetration tests on AI surfaces (prompt injection, indirect injection via retrieved content, tool abuse) require specialised testers. Ask whether AI-specific testing was in scope and which threats were tested.
Data processing agreement (DPA) — the legal instrument that governs personal data handling. Required under GDPR if personal data is processed. Should explicitly cover AI use of the data, training data exclusion, and sub-processor commitments.
Model card or system card — vendor-provided documentation of the AI system. Quality varies. A useful model card names the base model, training data sources, known limitations, evaluation results, and intended use cases. Generic marketing pages are not model cards.

Each gap in this evidence list is itself recordable evidence. A vendor that declines to provide a SOC 2 report has provided you with an evidence gap, which becomes a control gap in your assessment, which informs the disposition.

Vendor AI Security Questionnaire

20 adversarial questions across 7 sections. Each question paired with required evidence and a follow-up if the vendor declines. Free download.

The vendor assessment questionnaire structure

A useful vendor AI questionnaire is adversarial in structure, not extractive. Extractive questionnaires ask the vendor to describe their AI feature. Adversarial questionnaires assume the description and then ask follow-up questions that test the description against documented evidence.

The questionnaire should produce a record covering at least seven sections: system description (what the AI feature does and how), data flows (what enters the model, what is returned, what is retained, where), model governance (who provides the model, how it is updated, how the vendor evaluates model changes),access controls (who at the vendor can see customer data and prompts),incident response (notification policy, disclosure commitments, evidence provided post-incident), sub-processor accountability (list with dates, notification on change, contractual flow-down), and re-assessment triggers (what vendor changes require you to re-review).

Each question is paired with an evidence requirement. “Does the vendor train on customer data?” is paired with “Provide the contractual clause that prohibits this.” A yes/no answer with no supporting evidence is recorded as an evidence gap.

EU AI Act implications for vendor procurement

Under the EU AI Act, the obligations attach to two roles: providers (who develop and place AI systems on the market) and deployers (who use AI systems in their operations). When you procure a vendor AI feature and use it in your workflow, you are a deployer. If the AI feature qualifies as a high-risk AI system under Annex III, you have deployer obligations even though you did not build the system.

Deployer obligations include: ensuring the system is used in accordance with the provider's instructions, monitoring the system in operation, maintaining logs where the provider has provided log functionality, and informing affected persons that they are subject to a high-risk AI system. None of these are satisfied by “the vendor is responsible”. They are your obligations.

For high-risk vendor systems, the deployer should retain a documented risk record (your own, separate from the provider's Article 9 risk management). The Drel vendor assessment record supports this — it documents your assessment of the system, the conditions under which you authorised its use, and the re-assessment triggers you have committed to.

Assessment disposition for vendor systems

A vendor system rarely receives an unconditional proceed disposition on first assessment. The typical outcome is a conditional clearance with explicit contractual requirements that must be met before production use. Examples of conditions that appear regularly:

Vendor must execute an updated DPA naming the AI feature explicitly and confirming the training data exclusion clause.
Vendor must commit in writing to a sub-processor change notification window of at least 30 days.
Vendor must provide a model card or system card before deployment. Generic FAQ pages are not acceptable.
Customer data classified as “restricted” under your data classification policy is excluded from the AI feature pending further assessment.
Re-assessment triggers: vendor changes the underlying model, adds a new sub-processor, changes the data retention period, or publicly discloses an AI-related incident.

These conditions belong in the disposition record, alongside an owner for each condition and a deadline. They convert “we did a security review” into “we authorised this vendor under these specific conditions, and here is the evidence”.

Frequently asked questions

What evidence can we demand from AI vendors?: At minimum: a data processing agreement that explicitly covers AI use of customer data, a sub-processor list with change notification, and a data retention policy. Better: a SOC 2 Type II report whose scope includes the AI feature, an ISO 27001 certificate, a model card, and a recent penetration test report. Best: an independent assessment of the AI feature against OWASP LLM Top 10 or OWASP Agentic Top 10, and a defined incident disclosure policy.
How does Drel handle vendor SaaS AI?: Drel reviews the vendor AI system based on what the vendor has documented and disclosed. It maps the documented architecture, declared data flows, and contractual control claims to required controls and produces a clearance decision with explicit conditions. Each evidence gap (something the vendor has not documented or evidenced) is recorded as a control gap in the disposition.
What if the vendor refuses to share design artifacts?: Refusal to share documentation is itself an evidence gap and goes into the assessment record. The conditional clearance can include a requirement that the vendor provide specific documentation before moving from pilot to production. A vendor that consistently refuses to provide documentation is producing a control gap that the AI Committee must explicitly accept as residual risk or reject as a blocker.
Does Drel certify vendors?: No. Drel produces a structured assessment of the vendor's system based on available documentation. The output is your record of having assessed the vendor — it documents what you reviewed, what you accepted, and what you required. Certification is a separate, accreditation-based process performed by audit bodies, not by review platforms.
What triggers a re-assessment of a vendor system?: The vendor changes their model or model version, adds training data sources, changes sub-processors, updates their data handling policy in ways that affect your use, has a publicly disclosed AI-related incident, or expands the scope of the AI feature (new data types, new use cases). Each trigger should be named in the original disposition with an owner.
How does the EU AI Act apply to vendor AI?: Under the EU AI Act, deployers (organisations using AI systems) have obligations distinct from providers (who built the system). For high-risk AI systems used in your operations, you retain deployer obligations including monitoring use, ensuring the system is used per provider instructions, and informing affected persons. The vendor AI assessment record supports the evidence for these obligations.

A note on scope: Drel reviews assessed systems against documented architecture, configuration and intent. It does not ingest live telemetry from production environments. Dispositions reflect the assessed system at the time of review and the re-assessment triggers that govern when the disposition must be revisited.