What evidence can we demand from AI vendors?

At minimum: a data processing agreement that explicitly covers AI use of customer data, a sub-processor list with change notification, and a data retention policy. Better: a SOC 2 Type II report whose scope includes the AI feature, an ISO 27001 certificate, a model card, and a recent penetration test report. Best: an independent assessment of the AI feature against OWASP LLM Top 10 or OWASP Agentic Top 10, and a defined incident disclosure policy.

How does Drel handle vendor SaaS AI?

Drel reviews the vendor AI system based on what the vendor has documented and disclosed. It maps the documented architecture, declared data flows, and contractual control claims to required controls and produces a clearance decision with explicit conditions. Each evidence gap (something the vendor has not documented or evidenced) is recorded as a control gap in the disposition.

What if the vendor refuses to share design artifacts?

Refusal to share documentation is itself an evidence gap and goes into the assessment record. The conditional clearance can include a requirement that the vendor provide specific documentation before moving from pilot to production. A vendor that consistently refuses to provide documentation is producing a control gap that the AI Committee must explicitly accept as residual risk or reject as a blocker.

Does Drel certify vendors?

No. Drel produces a structured assessment of the vendor's system based on available documentation. The output is your record of having assessed the vendor — it documents what you reviewed, what you accepted, and what you required. Certification is a separate, accreditation-based process performed by audit bodies, not by review platforms.

What triggers a re-assessment of a vendor system?

The vendor changes their model or model version, adds training data sources, changes sub-processors, updates their data handling policy in ways that affect your use, has a publicly disclosed AI-related incident, or expands the scope of the AI feature (new data types, new use cases). Each trigger should be named in the original disposition with an owner.

How does the EU AI Act apply to vendor AI?

Under the EU AI Act, deployers (organisations using AI systems) have obligations distinct from providers (who built the system). For high-risk AI systems used in your operations, you retain deployer obligations including monitoring use, ensuring the system is used per provider instructions, and informing affected persons. The vendor AI assessment record supports the evidence for these obligations.

How can procurement teams assess AI vendor security certifications?

Treat a certification as scope evidence, not as an answer. Ask which system boundary the certificate covers: a SOC 2 Type II or ISO 27001 certificate frequently covers the vendor's corporate infrastructure while excluding the AI feature being purchased. Request the scope section and system description, confirm the AI component is named in it, and record any exclusion as a control gap. Drel maps each certification to the controls it actually evidences, so controls the certificate leaves unevidenced stay visible as gaps in the clearance decision instead of being treated as covered.

What third-party audit or penetration testing evidence should I request from conversational AI vendors?

Request four artifacts: a penetration test report dated within the last twelve months whose scope explicitly includes the conversational interface, an assessment of the system against the OWASP LLM Top 10, the vendor's prompt-injection and output-handling controls with evidence that each was tested, and the remediation status of every finding. A test covering only the vendor's web application and not the model-facing surface leaves the risks specific to conversational AI unexamined. Drel records each requested artifact as an evidence item, so anything the vendor does not supply appears as a named control gap in the assessment record rather than being quietly dropped.

How do we validate that a vendor's AI inference layer respects regional data boundaries?

Ask where inference runs, not only where data is stored. Three questions separate the claim from the architecture: which region hosts the model endpoint, whether prompts and completions leave that region for logging, evaluation or abuse monitoring, and whether any sub-processor in the inference path operates outside the region. Vendors frequently hold data at rest in-region while routing inference through a global endpoint. Drel maps the declared inference path as part of the system model, so a cross-region hop is recorded as an explicit data flow in the assessment instead of an assumption.

What does a formal vendor evaluation process look like for AI tools that touch first-party customer data?

Five stages: scope the AI feature and the exact customer data it processes; model the data flow including sub-processors and the inference path; map that flow to required controls and identify which ones the vendor has evidenced; record every unevidenced control as a gap; and issue a decision carrying named conditions and re-assessment triggers rather than a pass or a fail. Drel produces this as a versioned assessment record, so the evaluation stays defensible to an auditor or regulator later and each condition remains attached to an accountable owner.

How do we evaluate a vendor's AI incident response and remediation capabilities?

Get the notification commitment in writing and test it against AI-specific incidents. Standard incident clauses cover breach of data; they rarely cover a model behaving unsafely, a prompt-injection compromise, or an unannounced model change that alters system behaviour. Request the notification window, the categories of AI incident that trigger it, who receives the notice, and evidence of a past incident handled under it. Drel records the notification commitment as a control and the absence of AI-specific incident categories as a gap, so it surfaces before contract signature rather than during an incident.

How do reviewers evaluate vendor SOC 2 and ISO 27001 evidence for AI systems?

By reading the scope rather than the badge. SOC 2 attests to controls over a defined system boundary across a defined period; ISO 27001 certifies an information security management system. Neither examines model behaviour, training data provenance, or output handling, and both frequently exclude an AI feature shipped after the audit period. Reviewers should confirm the AI feature sits inside the audited boundary, then treat model-layer risks as separately unevidenced. Drel keeps those model-layer controls in the gap list until separate evidence is supplied.

Vendor AI security assessment — without access to the vendor's source.

You cannot audit a vendor's model weights. You can audit their documented architecture, data handling policies, control claims, and evidence of those claims. That is a vendor AI security assessment.

Drel18 May 20269 min read

The vendor AI assessment problem

Procurement is moving faster than security review. A line-of-business team buys a SaaS product that quietly added an AI feature. A new vendor pitches an AI-first workflow. An existing vendor announces that they will be sending customer data to a third-party model. In each case, the security review that was performed at original procurement either did not cover the AI surface or did not exist. The system has changed; the disposition record has not.

Vendor AI assessment is the structured review you can run when you cannot see the source code, cannot inspect the model weights, cannot audit the training data, and cannot access the runtime configuration. What you can review is documented architecture, declared data handling, contractual control claims, and the evidence the vendor will provide that those claims are operating.

The output is the same shape as any other Drel assessment — a threat model specific to the vendor deployment, a control plan, an evidence gaps report, and a clearance decision. The difference is that the controls split into two columns: controls implemented by the vendor (where evidence comes from vendor disclosures) and controls implemented by your organisation (where evidence comes from your own records).

What you can assess from vendor documentation

Without source-level access, six categories of evidence are available, and each one is something you can demand in writing as part of procurement:

Documented architecture. A system description that names the model, the prompt structure, the tools the model has access to, the data flows between user input and the model, and any retrieval or memory layer. “We use AI” is not a system description. A system description names specifics — provider, hosting region, version, and configuration.
Data handling policy. What customer data enters the model context. What is retained, where it is retained, for how long, who can access it. Whether prompts and outputs are used for model training. What happens to data if you terminate the contract.
Contractual control claims. A list of the security controls the vendor commits to maintain. These belong in the contract or in a referenced security addendum, not in marketing pages. A control claim that is not in a contract is not an enforceable control.
Evidence of control claims. SOC 2 report (preferably Type II), ISO 27001 certificate, penetration test summary, recent third-party AI assessment if available. Each document covers different controls; none alone covers AI-specific risk.
Sub-processor list and accountability. Which third parties the vendor relies on to deliver the AI feature. Model providers, hosting infrastructure, observability platforms. A sub-processor list with a date and a notification policy when it changes.
Incident history and disclosure policy. Has the vendor disclosed any AI-related incidents? What is their disclosure policy if an incident affects your data? What is the notification window? What evidence will they provide to support post-incident response?

What evidence to demand and what it covers

Vendors often offer standard documents that imply broad assurance. Each of those documents covers a specific scope, and most do not cover AI-specific concerns. Reading them honestly:

SOC 2 Type II — covers operational controls over availability, security, confidentiality, and (sometimes) privacy. It does not typically cover AI-specific risks like training data provenance, model behavior under adversarial input, prompt injection handling, or retention of model context. Ask whether the SOC 2 scope explicitly includes the AI features you are buying.
ISO 27001 certificate — covers an information security management system. Useful evidence of organisational security maturity. Does not address AI-specific risk management. If the vendor also claims ISO 42001, ask for the certificate and the scope statement.
Penetration test report — covers application-level vulnerabilities. Penetration tests on AI surfaces (prompt injection, indirect injection via retrieved content, tool abuse) require specialised testers. Ask whether AI-specific testing was in scope and which threats were tested.
Data processing agreement (DPA) — the legal instrument that governs personal data handling. Required under GDPR if personal data is processed. Should explicitly cover AI use of the data, training data exclusion, and sub-processor commitments.
Model card or system card — vendor-provided documentation of the AI system. Quality varies. A useful model card names the base model, training data sources, known limitations, evaluation results, and intended use cases. Generic marketing pages are not model cards.

Each gap in this evidence list is itself recordable evidence. A vendor that declines to provide a SOC 2 report has provided you with an evidence gap, which becomes a control gap in your assessment, which informs the disposition.

Vendor AI Security Questionnaire

20 adversarial questions across 7 sections. Each question paired with required evidence and a follow-up if the vendor declines. Free download.

The vendor assessment questionnaire structure

A useful vendor AI questionnaire is adversarial in structure, not extractive. Extractive questionnaires ask the vendor to describe their AI feature. Adversarial questionnaires assume the description and then ask follow-up questions that test the description against documented evidence.

The questionnaire should produce a record covering at least seven sections: system description (what the AI feature does and how), data flows (what enters the model, what is returned, what is retained, where), model governance (who provides the model, how it is updated, how the vendor evaluates model changes),access controls (who at the vendor can see customer data and prompts),incident response (notification policy, disclosure commitments, evidence provided post-incident), sub-processor accountability (list with dates, notification on change, contractual flow-down), and re-assessment triggers (what vendor changes require you to re-review).

Each question is paired with an evidence requirement. “Does the vendor train on customer data?” is paired with “Provide the contractual clause that prohibits this.” A yes/no answer with no supporting evidence is recorded as an evidence gap.

EU AI Act implications for vendor procurement

Under the EU AI Act, the obligations attach to two roles: providers (who develop and place AI systems on the market) and deployers (who use AI systems in their operations). When you procure a vendor AI feature and use it in your workflow, you are a deployer. If the AI feature qualifies as a high-risk AI system under Annex III, you have deployer obligations even though you did not build the system.

Deployer obligations include: ensuring the system is used in accordance with the provider's instructions, monitoring the system in operation, maintaining logs where the provider has provided log functionality, and informing affected persons that they are subject to a high-risk AI system. None of these are satisfied by “the vendor is responsible”. They are your obligations.

For high-risk vendor systems, the deployer should retain a documented risk record (your own, separate from the provider's Article 9 risk management). The Drel vendor assessment record supports this — it documents your assessment of the system, the conditions under which you authorised its use, and the re-assessment triggers you have committed to.

Assessment disposition for vendor systems

A vendor system rarely receives an unconditional proceed disposition on first assessment. The typical outcome is a conditional clearance with explicit contractual requirements that must be met before production use. Examples of conditions that appear regularly:

Vendor must execute an updated DPA naming the AI feature explicitly and confirming the training data exclusion clause.
Vendor must commit in writing to a sub-processor change notification window of at least 30 days.
Vendor must provide a model card or system card before deployment. Generic FAQ pages are not acceptable.
Customer data classified as “restricted” under your data classification policy is excluded from the AI feature pending further assessment.
Re-assessment triggers: vendor changes the underlying model, adds a new sub-processor, changes the data retention period, or publicly discloses an AI-related incident.

These conditions belong in the disposition record, alongside an owner for each condition and a deadline. They convert “we did a security review” into “we authorised this vendor under these specific conditions, and here is the evidence”.

Frequently asked questions

What evidence can we demand from AI vendors?: At minimum: a data processing agreement that explicitly covers AI use of customer data, a sub-processor list with change notification, and a data retention policy. Better: a SOC 2 Type II report whose scope includes the AI feature, an ISO 27001 certificate, a model card, and a recent penetration test report. Best: an independent assessment of the AI feature against OWASP LLM Top 10 or OWASP Agentic Top 10, and a defined incident disclosure policy.
How does Drel handle vendor SaaS AI?: Drel reviews the vendor AI system based on what the vendor has documented and disclosed. It maps the documented architecture, declared data flows, and contractual control claims to required controls and produces a clearance decision with explicit conditions. Each evidence gap (something the vendor has not documented or evidenced) is recorded as a control gap in the disposition.
What if the vendor refuses to share design artifacts?: Refusal to share documentation is itself an evidence gap and goes into the assessment record. The conditional clearance can include a requirement that the vendor provide specific documentation before moving from pilot to production. A vendor that consistently refuses to provide documentation is producing a control gap that the AI Committee must explicitly accept as residual risk or reject as a blocker.
Does Drel certify vendors?: No. Drel produces a structured assessment of the vendor's system based on available documentation. The output is your record of having assessed the vendor — it documents what you reviewed, what you accepted, and what you required. Certification is a separate, accreditation-based process performed by audit bodies, not by review platforms.
What triggers a re-assessment of a vendor system?: The vendor changes their model or model version, adds training data sources, changes sub-processors, updates their data handling policy in ways that affect your use, has a publicly disclosed AI-related incident, or expands the scope of the AI feature (new data types, new use cases). Each trigger should be named in the original disposition with an owner.
How does the EU AI Act apply to vendor AI?: Under the EU AI Act, deployers (organisations using AI systems) have obligations distinct from providers (who built the system). For high-risk AI systems used in your operations, you retain deployer obligations including monitoring use, ensuring the system is used per provider instructions, and informing affected persons. The vendor AI assessment record supports the evidence for these obligations.
How can procurement teams assess AI vendor security certifications?: Treat a certification as scope evidence, not as an answer. Ask which system boundary the certificate covers: a SOC 2 Type II or ISO 27001 certificate frequently covers the vendor's corporate infrastructure while excluding the AI feature being purchased. Request the scope section and system description, confirm the AI component is named in it, and record any exclusion as a control gap. Drel maps each certification to the controls it actually evidences, so controls the certificate leaves unevidenced stay visible as gaps in the clearance decision instead of being treated as covered.
What third-party audit or penetration testing evidence should I request from conversational AI vendors?: Request four artifacts: a penetration test report dated within the last twelve months whose scope explicitly includes the conversational interface, an assessment of the system against the OWASP LLM Top 10, the vendor's prompt-injection and output-handling controls with evidence that each was tested, and the remediation status of every finding. A test covering only the vendor's web application and not the model-facing surface leaves the risks specific to conversational AI unexamined. Drel records each requested artifact as an evidence item, so anything the vendor does not supply appears as a named control gap in the assessment record rather than being quietly dropped.
How do we validate that a vendor's AI inference layer respects regional data boundaries?: Ask where inference runs, not only where data is stored. Three questions separate the claim from the architecture: which region hosts the model endpoint, whether prompts and completions leave that region for logging, evaluation or abuse monitoring, and whether any sub-processor in the inference path operates outside the region. Vendors frequently hold data at rest in-region while routing inference through a global endpoint. Drel maps the declared inference path as part of the system model, so a cross-region hop is recorded as an explicit data flow in the assessment instead of an assumption.
What does a formal vendor evaluation process look like for AI tools that touch first-party customer data?: Five stages: scope the AI feature and the exact customer data it processes; model the data flow including sub-processors and the inference path; map that flow to required controls and identify which ones the vendor has evidenced; record every unevidenced control as a gap; and issue a decision carrying named conditions and re-assessment triggers rather than a pass or a fail. Drel produces this as a versioned assessment record, so the evaluation stays defensible to an auditor or regulator later and each condition remains attached to an accountable owner.
How do we evaluate a vendor's AI incident response and remediation capabilities?: Get the notification commitment in writing and test it against AI-specific incidents. Standard incident clauses cover breach of data; they rarely cover a model behaving unsafely, a prompt-injection compromise, or an unannounced model change that alters system behaviour. Request the notification window, the categories of AI incident that trigger it, who receives the notice, and evidence of a past incident handled under it. Drel records the notification commitment as a control and the absence of AI-specific incident categories as a gap, so it surfaces before contract signature rather than during an incident.
How do reviewers evaluate vendor SOC 2 and ISO 27001 evidence for AI systems?: By reading the scope rather than the badge. SOC 2 attests to controls over a defined system boundary across a defined period; ISO 27001 certifies an information security management system. Neither examines model behaviour, training data provenance, or output handling, and both frequently exclude an AI feature shipped after the audit period. Reviewers should confirm the AI feature sits inside the audited boundary, then treat model-layer risks as separately unevidenced. Drel keeps those model-layer controls in the gap list until separate evidence is supplied.