> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/planning-and-decision-guides/ai-testing-vendor-security-evidence.md).

# AI Testing Vendor Security Evidence

Request verifiable evidence for AI data flows, model providers, repositories, credentials, access controls, retention, deletion, and revocation.

> **Editorial owner:** ContextQA Documentation · **Last updated:** September 12, 2026

## Quick answer

Ask the vendor to map each data category through collection, storage, model processing, artifact generation, retention, deletion, and support access. Then separate verified product behavior, configurable controls, contractual commitments, and open exceptions. A certification logo or generic security page cannot answer your workflow-specific questions by itself.

## Scope the exact workflow

Record the deployment, test target, integrations, AI client, model path, regions, and users in scope. Security answers for a browser-only public demo do not automatically cover private repositories, ticket text, screenshots, network logs, database steps, or production-like test data.

## Inventory data and destinations

```mermaid
flowchart LR
    A[Repository and requirements] --> B[Testing platform]
    C[Test data and credentials] --> B
    B --> D[Execution environment]
    B --> E[Model or subprocessors]
    B --> F[Logs, artifacts, and evidence]
    D --> F
```

| Data category       | Example                                  | Sent where? | Stored where? | Retention/deletion owner |
| ------------------- | ---------------------------------------- | ----------- | ------------- | ------------------------ |
| Requirements        | Tickets, specs, meeting notes            |             |               |                          |
| Source context      | Repository files and diffs               |             |               |                          |
| Credentials         | Test accounts, tokens, vault references  |             |               |                          |
| Application data    | Synthetic or customer-like records       |             |               |                          |
| Execution artifacts | Screenshots, video, HAR, console, traces |             |               |                          |
| AI inputs/outputs   | Prompts, context, model responses        |             |               |                          |
| Support data        | Diagnostics and uploaded evidence        |             |               |                          |

Require separate answers for the platform, each downstream model provider, subprocessors, plugins/tools, and customer-controlled infrastructure.

## Request evidence by control area

### Identity and least privilege

* SSO and lifecycle-management options relevant to your plan;
* workspace roles and separation of read, run, change, delete, and release actions;
* service-account ownership and rotation;
* support and administrative access controls;
* audit records for sensitive actions; and
* immediate revocation procedure.

### Repository and integrations

* exact scopes requested from source control and ticketing systems;
* whether full repository contents or selected artifacts are processed;
* webhook validation and secret storage;
* behavior after an integration is disconnected; and
* evidence that a read-only evaluation cannot mutate source systems.

### Test data and credentials

* approved secret sources and masking behavior;
* whether screenshots, HAR files, console logs, traces, or videos can contain sensitive data;
* redaction controls and documented limitations;
* synthetic-data and reset recommendations; and
* incident procedure for exposed credentials.

### AI and model providers

* which provider and endpoint handles each workflow;
* retention and training policy at the endpoint level;
* routing or fallback behavior;
* regional-processing options;
* customer controls over model/provider selection; and
* whether plugins or tools have separate data policies.

### Retention, deletion, and portability

* default and configurable retention by artifact;
* deletion behavior, backup implications, and verification;
* offboarding and account termination process;
* exportable tests/evidence and residual dependencies; and
* legal or contractual retention exceptions.

## Maintain an evidence index

| Requirement                                | Evidence supplied                   | Version/date | Verified by | Status                          |
| ------------------------------------------ | ----------------------------------- | ------------ | ----------- | ------------------------------- |
| Runs use approved test data only           | Configuration + representative run  |              |             | Met / Config / Exception / Open |
| Repository access is least privilege       | OAuth scope and observed access     |              |             |                                 |
| Model path meets retention policy          | Endpoint-specific policy and config |              |             |                                 |
| Credentials can be revoked                 | Tested procedure                    |              |             |                                 |
| Required artifacts can be deleted/exported | Demonstration or contract           |              |             |                                 |

Do not copy confidential agreements into public documentation. Link the approved internal location and expose only the status and owner to the evaluation team.

## Define blocking exceptions

An exception needs affected data, likelihood, impact, compensating control, approver, expiry, and remediation owner. Stop the pilot if a required data flow is unknown, a critical secret cannot be protected, revocation cannot be performed, or policy depends on an unverified endpoint assumption.

## Frequently asked questions

### Is a SOC 2 report enough evidence for an AI testing vendor?

No. It may support the review, but it does not by itself answer workflow-specific questions about repository scope, test credentials, model providers, subprocessors, retention, deletion, evidence access, or incident handling. Map each required control to current evidence.

### Does the platform’s policy automatically cover its model providers?

Not necessarily. Record every destination that may receive QA context and verify the provider, endpoint, retention, training, region, and fallback behavior for the route actually used. Treat an unresolved downstream path as unresolved risk.

### Which security gaps should block a pilot?

Block or restrict the pilot when required data classification, access control, credential handling, deletion, provider routing, auditability, or incident-response conditions are not satisfied. A safe synthetic-data pilot may still be possible if its boundary is explicit and enforced.

## Related documentation

* [Roles & Permissions](/administration/roles-and-permissions.md)
* [API Tokens](/administration/api-tokens.md)
* [MCP Server Overview](/mcp-server/overview.md)
* [OpenRouter Controls for QA Context](/planning-and-decision-guides/openrouter-controls-for-qa-context.md)

{% hint style="info" %}
[**Request the current ContextQA evidence for your scoped workflow →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=vendor_security_evidence). Treat unresolved requirements as open—not implicitly satisfied.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/planning-and-decision-guides/ai-testing-vendor-security-evidence.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.