> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/evaluation-and-governance/ai-testing-pilot-scorecard.md).
# 30-Day AI Testing Pilot Scorecard
Compare an AI testing candidate with your current approach using equivalent scenarios, seeded failures, effort records, disqualifiers, and decision gates.
## Quick answer
Run an AI testing pilot against a versioned application fixture and a comparable baseline. Use the same workflows, assertions, data, browsers, and seeded defects for both approaches; record human effort and incomplete runs as well as successes. Agree on coverage, reliability, ownership, integration, and portability gates before the first candidate run. At day 30, make an adopt, revise, extend, or stop decision from the evidence packet—not from a demo or a generated-test count.
## What this guide produces
This guide combines the pilot plan and its scorecard so the team does not maintain two competing evaluation documents. Copy the tables into your issue tracker, design document, or evaluation repository.
The result is a decision packet containing:
* a frozen baseline and candidate configuration;
* a representative workflow and assertion inventory;
* a known-defect test set;
* weekly execution and operating-effort ledgers;
* coverage and reliability disqualifiers;
* a portability and ownership review; and
* a signed decision record.
{% hint style="info" %}
The templates below are team-defined evaluation controls. They are not automatic ContextQA API fields or guaranteed product outcomes.
{% endhint %}
## Start with a fair comparison
“AI-generated tests versus our existing suite” is not yet a fair comparison. The candidate and baseline must attempt the same reader-visible job.
| Control | Baseline | Candidate |
| ------------------- | -------------------------------------------------- | -------------------------------------------------- |
| Application | Same commit or immutable build | Same commit or immutable build |
| Environment | Same service dependencies and network path | Same service dependencies and network path |
| Data | Same reset procedure and accounts | Same reset procedure and accounts |
| Workflows | Same critical user journeys | Same critical user journeys |
| Assertions | Same business outcomes | Same business outcomes |
| Browsers or devices | Same required matrix | Same required matrix |
| Failures | Same seeded defects | Same seeded defects |
| Human work | Creation, review, triage, and maintenance included | Creation, review, triage, and maintenance included |
If one approach cannot represent an assertion or environment constraint, record that as a result. Do not silently remove the constraint from both cohorts.
### Freeze the evaluation fixture
Record the following before the first run:
```
Application repository:
Commit or immutable build ID:
Fixture version:
Environment name and revision:
Seed-data revision:
Required browsers/devices:
Baseline framework and version:
Candidate configuration revision:
Evaluation owner:
Decision date:
```
If the baseline uses Playwright, keep its test plan, fixtures, and assertions under version control. Playwright's official test-agent workflow similarly separates a human-readable plan, a seed test, and generated tests; that structure is a useful comparison boundary even if you do not use its agents. See [Playwright Test Agents](https://playwright.dev/docs/test-agents).
## Select representative scenarios
Choose a small set that exposes operational differences. Ten carefully selected workflows are usually more informative than a large set of happy paths.
Include at least:
1. A revenue or activation path.
2. An authenticated flow with role or permission boundaries.
3. A workflow with dynamic UI state.
4. A negative path with a business assertion.
5. A data-dependent or API-backed flow.
6. A maintenance case based on a real, nonfunctional UI change.
7. A failure with diagnostic evidence requirements.
8. A workflow that must run in CI.
For every scenario, complete this contract:
| Field | Example format |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| Scenario ID | `PILOT-04` |
| User risk | A viewer must not approve an invoice |
| Preconditions | Viewer account, one pending invoice |
| Steps | Login as viewer, attempt approval through the in-scope UI/API paths, inspect invoice state |
| Required assertion | Viewer approval is denied and invoice state remains unchanged; check the UI and, where in scope, the corresponding API |
| Evidence | Role, relevant assertion result, observed state, and final run record |
| Baseline test | Repository path and test name |
| Candidate test | ContextQA test-case reference |
| Data reset | Fixture command or named data profile |
| Reviewer | Named engineering or QA reviewer |
## Seed failures before measuring reliability
A tool that reports green on an always-green application has not proved much. Introduce controlled defects in a disposable fixture or branch. Never seed a defect in a shared production environment.
Use failures that distinguish false passes, diagnostic quality, and environment handling:
| Seed | Expected observation |
| -------------------------------------------------------- | -------------------------------------------------------------- |
| Change a required confirmation message | The business assertion fails |
| Remove an authorization check in the fixture | The negative permission test fails |
| Return a controlled server error | The run fails and preserves useful diagnostics |
| Delay a dependency beyond the agreed timeout | The run becomes a timeout or incomplete result, not a pass |
| Rename or reposition a control without changing behavior | The maintenance case records whether intervention was required |
| Point the runner at the wrong build | The gate detects or exposes the build-identity mismatch |
Record the seed commit, the expected failing scenario, the expected reason, and the cleanup commit. If a seeded defect passes, stop expansion until the missing assertion or false-pass behavior is understood.
## Interpret control outcomes before scoring
Keep the execution status separate from the business conclusion. For a permission rule, use the following control matrix:
| Check | Expected application behavior | Expected test verdict |
| --------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------ |
| Allowed editor action | The permitted change succeeds and persists | Pass |
| Forbidden viewer action | The change is denied and stored data remains unchanged | Pass, because the test asserts denial |
| Deliberately weakened permission rule in the disposable fixture | The forbidden change becomes possible | Fail for the relevant permission assertion |
| Authentication or environment failure before the action | The business rule was not exercised | Unknown for that rule |
A failed run is not sufficient evidence that a seeded defect was detected. Confirm that the relevant assertion ran and failed for the intended reason. Restore the fixture and rerun the original controls afterward.
A repeated pass can still check the wrong outcome. A variable result can come from the application, test, environment, data or dependency. Preserve first-attempt outcomes and report retries separately. Label the cause unresolved until evidence distinguishes a real intermittent product failure from suspected test flakiness.
Use **Pass** when the agreed condition is observed, **Fail** when evidence contradicts it, and **Unknown** when the condition was not exercised or evidence cannot support a conclusion. An unknown mandatory gate holds the adoption decision; it is not a conditional pass and cannot be offset by a weighted score.
## Define disqualifiers and remediation windows
Write disqualifiers before candidate testing starts.
| Disqualifier | Immediate response | Remediation window |
| ------------------------------------------------- | -------------------------------- | ----------------------------------------------------- |
| A critical workflow has no enforceable assertion | Pause that workflow | Owner and date required before resuming |
| A seeded security or permission defect passes | Stop the pilot expansion | Root cause and rerun required |
| The executed build cannot be identified | Mark the result `Unknown` | Fix identity propagation before gating |
| A timed-out or canceled run is reported as passed | Keep all gates advisory | Correct state mapping and rerun the acceptance matrix |
| Required evidence is absent or inaccessible | Do not count the run as verified | Restore evidence or document an approved substitute |
| Export or ownership claims cannot be demonstrated | Record as a contract dependency | Complete the portability test before purchase |
Do not average a disqualifier away. A pilot can be fast and inexpensive while still being unsafe for its intended release decision.
## Use a four-week operating plan
### Week 1: baseline and configuration
* Freeze the fixture, scenario contracts, and seeded defects.
* Run the current approach three times under normal conditions.
* Configure the candidate without changing the acceptance criteria.
* Record initial setup and review time by role.
### Week 2: creation and first execution
* Create or migrate the candidate tests.
* Review every required assertion before running.
* Execute the normal and seeded-failure cohorts.
* Classify missing coverage, false passes, false failures, and incomplete runs.
### Week 3: maintenance and integration
* Apply the planned nonfunctional UI change.
* Measure the intervention needed in each approach.
* Exercise the CI path in advisory mode.
* Test failure, timeout, cancellation, retry, and wrong-build behavior with the [CI acceptance runbook](/evaluation-and-governance/ci-integration-acceptance-runbook.md).
### Week 4: repeated runs and decision
* Repeat the comparable suite under normal team conditions.
* Complete the ownership and portability checks.
* Review the evidence with engineering, QA, and security stakeholders.
* Decide: adopt, revise, extend for one named unknown, or stop.
## Record every run
Use one row per attempt, including attempts that never reached a terminal test result.
| Field | Value |
| -------------------------------- | --------------------------------------------------------- |
| Run date and cohort | |
| Application commit/build | |
| Environment and data revision | |
| Baseline or candidate | |
| Scenario IDs | |
| Runner/browser/device | |
| Start and finish time | |
| Terminal state | Pass / Fail / Incomplete / Canceled / Timed out / Unknown |
| Expected seeded failures | |
| Observed failures | |
| False pass or false failure | |
| Evidence reference | |
| Human triage minutes | |
| Missing or inaccessible evidence | |
| Reviewer and disposition | |
For ContextQA runs, use the saved plan and [test results](/reporting/test-results.md) rather than reconstructing results from CI console text.
## Measure operating effort
Separate elapsed machine time from human time. Both matter, but they answer different questions.
| Activity | Role | Baseline minutes | Candidate minutes | Notes/evidence |
| ----------------------------- | ---- | ---------------: | ----------------: | -------------- |
| Initial environment setup | | | | |
| Test creation or migration | | | | |
| Assertion review | | | | |
| Routine execution supervision | | | | |
| Failure triage | | | | |
| Test repair or maintenance | | | | |
| CI integration upkeep | | | | |
| Evidence packaging | | | | |
| Security and access review | | | | |
Avoid turning a one-month observation into an annual savings claim without stating the model. If you estimate annual impact, show the run frequency, labor-rate range, adoption ramp, and uncertainty separately.
## Calculate cost per usable verdict
“Cost per test” rewards cheap output even when the test is not reviewed, cannot fail, or never reaches a usable result. Use a denominator that reflects the decision your team consumes.
Define a **usable verdict** as a result that:
* ran the reviewed scenario and assertions;
* is tied to the intended application build, environment, and test revision;
* reached an accepted terminal state;
* includes the required evidence;
* was reviewed under the pilot protocol; and
* is not invalidated by missing coverage or an unresolved setup problem.
```
cost per usable verdict =
(platform + model + infrastructure + human review + triage + maintenance cost)
/ accepted usable verdicts
```
Record the calculation inputs:
| Field | Value |
| --------------------------------------------------------------- | ----- |
| Actual usage period and timezone | |
| Task/scenario cohort and sample size | |
| Total attempts | |
| Passing usable verdicts | |
| Failing usable verdicts | |
| Incomplete, canceled, timed-out, or invalid attempts | |
| Platform cost allocated to cohort | |
| Model input/output/cache/request cost, if separately observable | |
| Browser/device/infrastructure cost | |
| Human creation, review, triage, and maintenance time | |
| Labor-rate range and source | |
| Quality threshold and reviewer | |
| Missing costs/unknowns | |
Include failed and retried attempts in the numerator. Do not count the same avoided labor in two savings categories. Keep machine elapsed time and human time separate.
### Compare model or configuration variants fairly
If the workflow allows a meaningful model or agent configuration comparison, use the same:
* tasks and application build;
* tools and permission boundary;
* context pack and prompt contract;
* expected artifacts and quality rubric;
* retry/stop policy;
* reviewer protocol; and
* minimum repeated-trial count.
| Variant | Exact model/client/tool versions | Date | Attempts | Usable verdicts | Fail/incomplete | Median human review | Direct cost | Known limitation |
| ------- | -------------------------------- | ---- | -------: | --------------: | --------------: | ------------------: | ----------: | ---------------- |
| | | | | | | | | |
Do not publish a universal “best model” conclusion from one task set. Report the tested scope, versions, date, sample size, failure samples, and uncertainty. Do not imply that ContextQA supports model switching unless that control is actually exposed and verified in the evaluated workflow.
## Score with gates, then weights
First evaluate the non-negotiable gates.
| Gate | Pass condition | Result | Evidence |
| ------------------ | --------------------------------------------------------- | ------ | -------- |
| Coverage integrity | Critical assertions are present and reviewed | | |
| Failure fidelity | Seeded defects fail for the intended reason | | |
| Execution identity | Commit/build, environment, and test version are traceable | | |
| State integrity | Incomplete states cannot become a passing gate | | |
| Security | Secrets and permissions meet the pilot policy | | |
| Portability | Export, dependencies, and ownership are demonstrated | | |
Only after the gates pass, use the weighted scorecard. Choose weights before scoring.
| Criterion | Weight | Baseline 1–5 | Candidate 1–5 | Evidence |
| -------------------------------- | -----: | -----------: | ------------: | -------- |
| Coverage and assertion quality | 25% | | | |
| Reliability and failure fidelity | 20% | | | |
| Diagnostic evidence | 15% | | | |
| Human operating effort | 15% | | | |
| CI and workflow fit | 10% | | | |
| Security and administration | 10% | | | |
| Portability and ownership | 5% | | | |
Define `1`, `3`, and `5` for every criterion so reviewers do not score from intuition. For example, a `5` for diagnostic evidence might require a failed step, expected-versus-observed result, screenshot, execution log, and stable run reference; a `3` might require manual correlation; a `1` might be a generic failure message without reproducible evidence.
## Clarify ownership before rollout
| Responsibility | ContextQA | Your team | Another system | Named owner |
| ------------------------------------ | --------- | --------- | -------------- | ----------- |
| Scenario and assertion approval | | | | |
| Test data reset | | | | |
| Environment availability | | | | |
| Browser/device coverage | | | | |
| CI trigger and required-check policy | | | | |
| Failure triage | | | | |
| Test maintenance approval | | | | |
| Evidence retention | | | | |
| Export and exit test | | | | |
This table exposes the work that demos often hide. “Managed” does not mean “ownerless.”
## Write the final decision record
```
Decision: Adopt / Revise / Extend / Stop
Decision owner:
Date:
Approved scope:
Evidence packet:
Gates passed:
Gates failed:
Weighted result, if used:
Observed advantages:
Observed limitations:
Unresolved unknowns:
Required remediation and owner:
Rollout boundary:
Rollback trigger:
Next review date:
Approvers:
```
Choose **Extend** only for a named unknown with a bounded test and deadline. It should not become an automatic extra month.
## Run the weekly quality review after the pilot
Once the workflow is in use, hold a 30-minute weekly review that turns evidence into one owned improvement. Do not build a dashboard until the team agrees on definitions and data sources.
### Agenda
1. **Prior action:** Was last week’s improvement completed, and did the acceptance check pass?
2. **Important changes:** Which releases or pull requests carried the highest user risk?
3. **Evidence state:** Which decisions were `Pass`, `Fail`, or `Unknown`, and why?
4. **Coverage gaps:** Which critical requirements lack reviewed assertions?
5. **Unreliable tests:** Which tests were retried, skipped, flaky, or excluded, and what coverage did that remove?
6. **Operating burden:** Where did creation, review, queueing, triage, or maintenance time accumulate?
7. **One decision:** Select one bottleneck, owner, due date, and acceptance check for the next week.
### Copyable scorecard
| Metric | Numerator | Denominator | Current week | Prior week | Evidence source | Owner |
| --------------------------------------- | ------------------------------------------ | -------------------------- | -----------: | ---------: | --------------- | ----- |
| PRs with a usable verdict | PRs meeting the verdict definition | In-scope PRs | | | | |
| Critical requirement assertion coverage | Critical criteria with reviewed assertions | In-scope critical criteria | | | | |
| First-attempt complete runs | Complete, non-retried first attempts | Started attempts | | | | |
| Unknown decision rate | Unknown verdicts | In-scope decisions | | | | |
| Retry/skip affected coverage | Required cases retried or skipped | Required cases | | | | |
| Median PR-to-verdict time | — | Usable-verdict PRs | | | | |
| Median human intervention | — | Usable verdicts | | | | |
| Open quality actions past due | Past-due actions | Open actions | | | | |
If a denominator or source is unavailable, write `Not collected`. A synthetic example can teach the format, but only reviewed production data should drive a leadership claim.
### Action record
```
Week ending:
Highest unresolved risk:
Evidence:
Decision:
Owner:
Due date:
Acceptance check:
Coverage affected while open:
Escalation trigger:
```
The review should end with one or two decisions, not a list of unowned observations.
## Common questions
Should the candidate create exactly the same implementation as the baseline?
No. Compare the same user risk, workflow, assertion, environment, and evidence requirement. The implementation may differ; silently weakening the outcome may not.
Can generated-test count be a success metric?
It can describe throughput, but not decision quality. Count reviewed assertions, seeded-defect detection, complete runs, diagnostic value, and human effort alongside generated tests.
What if a product capability is unavailable in our workspace?
Record it as unavailable for the tested configuration. Do not score a roadmap statement as observed behavior. If it is essential, make enablement and a rerun a documented prerequisite.
## Related documentation
* [Evaluation and governance](/evaluation-and-governance/evaluation.md)
* [Running tests](/execution/running-tests.md)
* [Test results](/reporting/test-results.md)
* [Interpret a bug reproduction against the reported behavior](/mcp-server/reproduce-bug-from-ticket.md#step-2-interpret-the-result-against-the-reported-behavior)
* [Migrate an existing suite](/migrate-to-contextqa/migrating-existing-tests.md)
* [Prepare a CI/CD quality gate](/getting-started/ci-cd-tutorial.md)
{% hint style="info" %}
**Ready to test the scorecard on one real release path?** [**Book a pilot-design session →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=ai_pilot_scorecard) — We will help define the baseline, seeded failures, and stop conditions before the first candidate run.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://learning.contextqa.com/evaluation-and-governance/ai-testing-pilot-scorecard.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.