> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/planning-and-decision-guides/ai-testing-vendor-demo-scorecard.md).

# AI Testing Vendor Demo Scorecard

Compare AI testing vendors with the same workflow, evidence rubric, integration checks, operating boundaries, and portability test.

> **Editorial owner:** ContextQA Documentation · **Last updated:** September 12, 2026

## Quick answer

Give every shortlisted vendor the same representative task, environment constraints, failure, and evidence requirements. Score only what is demonstrated or supported by current documentation. Mark everything else **not demonstrated**—never infer a capability from a slide or adjacent feature.

## Define the pilot-qualification decision

The scorecard answers one question: **Which vendor, if any, earns a hands-on pilot on our required workflow?** It is not a final procurement decision and should not reward polished presentation quality.

Record before the first demo:

```
Application and non-production environment:
Workflow and critical business outcome:
Known failure or reversible seeded defect:
Browser/device/API requirements:
Identity, tenant, and test-data constraints:
CI and issue-tracker requirements:
Evidence and retention requirements:
Portability requirement:
Reviewers and conflicts of interest:
Decision date:
```

## Use one common task

```mermaid
flowchart LR
    A[One representative workflow] --> B[Same data and constraints]
    B --> C[Vendor demonstrations]
    C --> D[Evidence review]
    D --> E{Pilot-worthy?}
    E -- Yes --> F[Bounded pilot]
    E -- No --> G[Stop or revisit requirements]
```

Ask every vendor to complete the same sequence:

1. Create or import a test for the selected workflow.
2. Show the exact assertions, data, and environment configuration.
3. Run the unchanged test against normal behavior.
4. Enable the seeded failure and run it again.
5. Diagnose the failed result from captured evidence.
6. Make one locator-only application change and show maintenance behavior.
7. Trigger or describe the verified CI path, including timeout and cancellation.
8. Export or hand off one usable test artifact and list residual dependencies.

If live execution is impossible, record that limitation. A prerecorded flow can demonstrate user experience, but it does not prove compatibility with your application.

## Score evidence, not adjectives

Use `0–3` for each criterion:

* `0` — unavailable or contradicted;
* `1` — documented claim, not demonstrated;
* `2` — demonstrated with limitations or missing evidence;
* `3` — demonstrated on the common task with reviewable evidence.

| Criterion                                                       | Weight | Score | Evidence or limitation |
| --------------------------------------------------------------- | -----: | ----: | ---------------------- |
| Required assertions detect the seeded failure                   |      5 |       |                        |
| Result identifies build, environment, data, and browser/device  |      5 |       |                        |
| Failure artifacts support diagnosis without immediate rerun     |      4 |       |                        |
| Roles, tenant boundaries, and secrets are controlled            |      4 |       |                        |
| CI handles pass, fail, timeout, cancellation, and retry visibly |      4 |       |                        |
| Maintenance preserves business intent                           |      4 |       |                        |
| Existing-suite migration has explicit loss and review controls  |      3 |       |                        |
| Export and exit dependencies are documented                     |      3 |       |                        |
| Required integrations are current and supported                 |      3 |       |                        |
| Human setup, review, and triage effort is measured              |      3 |       |                        |

Do not let a high weighted total override a failed security, assertion-integrity, or build-identity gate.

## Ask questions a scripted demo cannot hide

* Which part was preconfigured before this meeting?
* What does the run report when the application is wrong but the locator still works?
* Which artifacts are unavailable for this run type or plan?
* What remains owned by our engineers after generation?
* What happens when the integration is unavailable or the result never arrives?
* Which permissions can create, run, edit, delete, or approve?
* Can we repeat the flow in our account during a pilot?
* What cannot be exported, and what is required to execute an export elsewhere?

## Decide the next step

Classify each vendor as:

* **Pilot** — critical gates passed and the common task is reproducible;
* **Clarify** — one bounded evidence gap has an owner and deadline;
* **Narrow** — viable only for a smaller scope;
* **Stop** — a disqualifier failed or the required workflow was not demonstrated.

Attach dated documentation links and result references. Recheck volatile integration, pricing, and availability claims before procurement.

## Frequently asked questions

### What workflow should we use for the vendor demo?

Choose a business-critical workflow your team understands and can safely reset. It should include meaningful assertions, realistic data, one expected failure, and at least one integration or operating constraint that matters in production.

### How many vendors should we compare?

Compare only the credible shortlist that survives your hard disqualifiers. A smaller comparison using the same fixture and evidence rubric produces a better decision than many shallow demonstrations with different scripts.

### What if a vendor will not run a negative-path test?

Record the capability as **not demonstrated**, not as passed. Decide whether the missing evidence is a pilot condition, a contractual requirement, or a reason to stop based on the risk of the workflow.

## Related documentation

* [30-Day AI Testing Pilot Scorecard](/evaluation-and-governance/ai-testing-pilot-scorecard.md)
* [Migration Planning & Acceptance Checklist](/migrate-to-contextqa/planning-checklist.md)
* [CI Integration Acceptance Runbook](/evaluation-and-governance/ci-integration-acceptance-runbook.md)
* [Clean-Environment Portability Test](/migrate-to-contextqa/migrating-existing-tests.md#run-a-clean-environment-portability-test)

{% hint style="info" %}
[**Use this scorecard in a ContextQA evaluation →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=vendor_demo_scorecard). Keep the same criteria for every shortlisted vendor.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/planning-and-decision-guides/ai-testing-vendor-demo-scorecard.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.