> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/evaluation-and-governance/evaluation.md).

# Evaluation & Governance Overview

Plan an evidence-led ContextQA evaluation with shared baselines, decision gates, ownership, portability checks, and release-quality proof.

## Quick answer

A useful test-automation evaluation asks whether a candidate improves release decisions on your application, with your workflows and constraints. Define the baseline, run the same representative scenarios, capture failures and operating effort, and agree on stop conditions before the pilot begins. Use the guides in this section to produce a decision packet that an engineering, QA, security, or procurement reviewer can inspect.

## What this section is for

Feature matrices are easy to fill out and hard to trust. Two tools may both claim cross-browser execution, CI integration, change-aware selection, and AI-assisted maintenance while placing very different work on your team.

These guides turn an evaluation into an observable system:

```mermaid
flowchart LR
    A[Production risk and constraints] --> B[Comparable baseline]
    B --> C[Representative pilot]
    C --> D[Execution evidence]
    D --> E[Operating burden]
    E --> F[Adopt, revise, or stop]
    F --> G[Signed decision record]
```

The goal is not to make every result look favorable. The goal is to make the decision explainable.

## Choose the guide that matches your decision

| Decision                                                        | Start here                                                                                                                       | Artifact you will produce                                                                                 |
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Which vendor should earn a hands-on pilot?                      | [AI testing vendor demo scorecard](/planning-and-decision-guides/ai-testing-vendor-demo-scorecard.md)                            | Common-task evidence matrix and pilot qualification decision                                              |
| What data and access evidence do security reviewers need?       | [AI testing vendor security evidence](/planning-and-decision-guides/ai-testing-vendor-security-evidence.md)                      | Evidence request, observed controls, and exception register                                               |
| Should we run a structured ContextQA pilot?                     | [AI testing pilot scorecard](/evaluation-and-governance/ai-testing-pilot-scorecard.md)                                           | Baseline, scenario set, weekly ledger, disqualifiers, and decision memo                                   |
| Can an AI-assisted PR gate produce reviewable release evidence? | [Agentic PR evidence checklist](/evaluation-and-governance/agentic-pr-evidence-checklist.md)                                     | Intent-to-test chain with build identity, run state, missing coverage, and owner                          |
| Is a reusable QA agent skill safe and reproducible?             | [QA agent skill validation](/evaluation-and-governance/qa-agent-skill-validation.md)                                             | Versioned skill, dependency, permission, negative-control, and repeatability record                       |
| Can we move from or coexist with a browser grid?                | [Browser-grid migration checklist](/evaluation-and-governance/browser-grid-migration-checklist.md)                               | Required-versus-observed matrix, overlap plan, and rollback criteria                                      |
| Is a CI quality gate ready to become required?                  | [CI integration acceptance runbook](/evaluation-and-governance/ci-integration-acceptance-runbook.md)                             | Pass, fail, timeout, cancellation, retry, and wrong-build acceptance record                               |
| Did a reported bug actually reproduce?                          | [Support-ticket reproduction workflow](/mcp-server/reproduce-bug-from-ticket.md)                                                 | Reviewed assertion, reproduced/not reproduced/blocked/inconclusive conclusion, evidence, and repair check |
| Does change-impact selection omit relevant tests?               | [PR Impact Analysis evaluation ledger](/ai-features/pr-impact-analysis.md#measure-selection-quality-with-a-run-ledger)           | Commit-specific selected-versus-broader-run comparison                                                    |
| Can we exit with usable assets and known ownership?             | [Clean-environment portability test](/migrate-to-contextqa/migrating-existing-tests.md#run-a-clean-environment-portability-test) | Export inventory, standalone run result, and residual-dependency register                                 |
| How should we migrate an existing automation portfolio?         | [Migration overview](/migrate-to-contextqa/migration.md)                                                                         | Source inventory, behavior packets, parallel comparison, acceptance, and rollback record                  |

## The evaluation contract

Write this one-page contract before anyone configures the candidate platform.

| Field              | What to record                                                              |
| ------------------ | --------------------------------------------------------------------------- |
| Decision           | The choice this pilot will inform, not a general desire to “try AI testing” |
| Decision owner     | One person accountable for the final adopt, revise, or stop call            |
| Technical owner    | Person responsible for environments, data, integrations, and execution      |
| Baseline window    | A representative period from the current process                            |
| Candidate window   | A comparable period using the candidate                                     |
| In-scope workflows | Named business journeys, APIs, browsers, devices, or release paths          |
| Out-of-scope work  | Anything the team will not infer from the pilot                             |
| Required evidence  | Run references, screenshots, logs, traces, defects, and effort records      |
| Disqualifiers      | Conditions that stop or narrow the evaluation                               |
| Decision date      | The date the evidence packet will be reviewed                               |

{% hint style="warning" %}
Do not change the baseline after seeing candidate results. If the scope must change, record the reason and evaluate the new cohort separately.
{% endhint %}

## Evidence hierarchy

Use the strongest available evidence and label weaker evidence honestly.

1. **Observed on your application:** a versioned run, trace, log, screenshot, defect, or reviewer decision.
2. **Reproduced in a representative fixture:** a controlled scenario with known assertions and seeded failures.
3. **Confirmed product behavior:** a documented ContextQA workflow verified in your workspace.
4. **External reference:** official documentation that defines the compared framework or CI behavior.
5. **Illustrative example:** a teaching example that must not be reported as a customer result.

Do not convert illustrative values into ROI claims. If a metric is missing, write `Not collected` and name the owner who will collect it.

## The minimum decision packet

At the end of the pilot, a reviewer should be able to answer:

* What application version, test version, environment, browser or device, and data state were exercised?
* Which workflows and assertions were included, and which important ones were omitted?
* Did the execution finish, or was it queued, canceled, timed out, or otherwise incomplete?
* Which failures were product defects, test defects, data or environment problems, or still unknown?
* How much human work was required to create, review, run, diagnose, and maintain the tests?
* What remains owned by scripts, a grid, a CI system, or another vendor?
* Can representative assets be exported and exercised outside the managed platform?
* What evidence supports adopting, revising, extending, or stopping?

## A practical decision rule

Use a small number of gates instead of a weighted score that can hide a fatal weakness.

| Gate               | Example pass condition                                                    | If it fails                                   |
| ------------------ | ------------------------------------------------------------------------- | --------------------------------------------- |
| Coverage integrity | Every critical workflow has a reviewed assertion or an explicit exception | Stop or remediate before measuring speed      |
| Build identity     | Every release decision points to the intended commit and environment      | Treat the result as unknown                   |
| Failure fidelity   | Seeded failures fail for the expected reason                              | Investigate false passes before expanding     |
| Operating burden   | Work is measured for both baseline and candidate                          | Do not publish an ROI conclusion              |
| Integration safety | Failure, timeout, cancellation, and retry behavior are deterministic      | Keep the gate advisory                        |
| Portability        | Exported assets and remaining dependencies are documented                 | Make the dependency an explicit contract term |

The scorecard may still include weighted criteria, but a high total must never override a failed disqualifier.

## Related documentation

* [Test Plans](/web-testing/test-plans.md)
* [Test Results](/reporting/test-results.md)
* [Prepare a CI/CD quality gate](/getting-started/ci-cd-tutorial.md)
* [ContextQA MCP server](/mcp-server/overview.md)
* [API tokens](/administration/api-tokens.md)

{% hint style="info" %}
**Want to build the evaluation around your application?** [**Book a technical evaluation session →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=evaluation_hub) — Bring one critical workflow, one recent failure, and your current CI path. We will help you define the evidence before discussing rollout.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/evaluation-and-governance/evaluation.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.