> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/evaluation-and-governance/agentic-pr-evidence-checklist.md).

# Agentic PR Evidence Checklist

Define the intent-to-test evidence chain, terminal states, ownership, missing coverage, and wrong-build controls for an agent-assisted PR review.

## Quick answer

An AI agent should not approve a pull request because it successfully called a test tool. A defensible gate must connect the change intent to affected paths, selected tests, reviewed assertions, the exact candidate build, a terminal execution result, and accessible evidence. Represent the outcome as `Pass`, `Fail`, or `Unknown`; keep the gate advisory until failure, timeout, cancellation, skipped-test, missing-coverage, and wrong-build cases behave as agreed.

## What this guide produces

Use this checklist when Claude Code, another MCP-compatible agent, or CI automation helps select or run ContextQA tests for a pull request. It produces a review record that answers five questions:

1. What change is being evaluated?
2. Why do these tests cover it?
3. Which exact application and test versions ran?
4. Did the execution finish with a trustworthy result?
5. Who owns missing coverage or remediation?

{% hint style="warning" %}
This is a team-defined governance pattern. It does not imply that ContextQA automatically enforces the gate, edits tests, or maps every field below into its API. The current [PR Impact Analysis](/ai-features/pr-impact-analysis.md) experience is advisory and read-only.
{% endhint %}

## Model the evidence chain

The gate is only as strong as its weakest link.

```mermaid
flowchart LR
    A[Change intent] --> B[Changed paths]
    B --> C[Affected behaviors]
    C --> D[Selected tests]
    D --> E[Reviewed assertions]
    E --> F[Candidate build]
    F --> G[Terminal run]
    G --> H[Evidence and decision]

    B -. unknown dependency .-> U[Unknown]
    D -. missing coverage .-> U
    F -. identity mismatch .-> U
    G -. incomplete state .-> U
```

A broken link is not automatically a failure in the product. It is an `Unknown` release decision until a reviewer expands coverage, verifies identity, or reruns the test.

## Define the three gate outcomes

| Outcome     | Meaning                                                                                                         | CI behavior                                                      |
| ----------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| **Pass**    | Required assertions ran against the intended build and all blocking results passed                              | May allow the protected action after acceptance testing          |
| **Fail**    | At least one required assertion reached a terminal failing result                                               | Block and assign a remediation owner                             |
| **Unknown** | The evidence is incomplete, mismatched, timed out, canceled, skipped unexpectedly, or missing required coverage | Block for a required gate; report neutrally for an advisory gate |

Avoid a binary mapping that treats everything other than an explicit failure as success. `Queued`, `running`, `canceled`, `timed out`, `partial`, and `no result` are not passing test outcomes.

## Step 1: Capture the change contract

Populate this record from the pull request and build system before asking an agent to select tests.

| Field                | Required value                                                                  |
| -------------------- | ------------------------------------------------------------------------------- |
| Repository           | Canonical repository identifier                                                 |
| Pull request         | Number and stable URL                                                           |
| Candidate commit     | Full commit SHA, not a moving branch name                                       |
| Base commit          | Merge base or comparison SHA                                                    |
| Build identifier     | Immutable artifact, deployment, or image ID                                     |
| Environment          | Named target and revision                                                       |
| Change intent        | One-paragraph author summary                                                    |
| Acceptance criteria  | Reviewable behavioral outcomes                                                  |
| Risk labels          | Auth, payments, data integrity, migration, permissions, or other relevant risks |
| Evaluation timestamp | When the evidence packet was assembled                                          |

If the pull request description conflicts with the diff, flag the mismatch. The change intent is an input, not proof.

## Check repository and requirement freshness

Cached context becomes unsafe when it describes a different commit, requirement revision, generated artifact, dependency graph, or test mapping from the one under review. Record provenance before reusing prior analysis.

| Context input                    | Required identity                                | Refresh trigger                                             | Unknown behavior                              |
| -------------------------------- | ------------------------------------------------ | ----------------------------------------------------------- | --------------------------------------------- |
| Repository snapshot              | Full commit SHA and included paths               | Head/base changes, force-push, submodule or lockfile change | Do not select tests from stale snapshot       |
| Requirements/acceptance criteria | Source URL/ID and revision timestamp             | Requirement edit, reopened decision, conflicting source     | Flag conflict; request owner decision         |
| Dependency/context graph         | Source inputs, build/repo revision, refresh time | Dependency, route, schema, or service map changes           | Expand fallback coverage                      |
| Test inventory/mapping           | Test IDs and last-reviewed revision              | Test created, retired, changed, or moved                    | Requery authoritative test source             |
| Environment/deployment           | Immutable build/deployment ID                    | Redeploy, rollback, feature-flag change                     | Mark build evidence `Unknown`                 |
| Agent context pack               | Version and source list                          | Any input above changes                                     | Rebuild pack; do not patch memory selectively |

### Freshness record

```
Analysis ID:
Repository head/base SHAs:
Requirement IDs/revisions:
Dependency-graph revision and generated-at:
Test-inventory generated-at:
Environment/build identity:
Agent context-pack version:
Known stale or unresolved inputs:
Decision: Reuse / Refresh / Unknown with broader fallback
Reviewer:
```

The policy above is team-defined unless your connected ContextQA workflow exposes and enforces the same metadata. Do not claim that the product invalidates stale context automatically without a verified run.

### Illustrative stale-versus-refreshed example

An earlier analysis at commit `A` selected checkout UI tests. Commit `B` adds a shared authorization-schema change after the cache was built. Reusing the commit-`A` mapping would omit a role-boundary test. A refreshed analysis adds the permission workflow; until that refresh finishes, the gate uses the broader regression fallback and reports `Unknown dependency` rather than `No impact`.

This example is illustrative. In a real review, preserve both reports and the actual selected-test difference.

## Step 2: Map paths to affected behavior

Start with the ContextQA [PR Impact Analysis](/ai-features/pr-impact-analysis.md) report when it is available. Review its scope, confidence, and `Partial` or `Superseded` state. Then add human knowledge that is not visible in the code graph.

| Changed path or dependency | Affected behavior            | Signal                 | Confidence | Reviewer note                   |
| -------------------------- | ---------------------------- | ---------------------- | ---------- | ------------------------------- |
| `services/permissions/*`   | Role-based approval controls | Structural mapping     | High       | Check inherited role behavior   |
| `ui/invoice/*`             | Invoice approval workflow    | Semantic match         | Medium     | Include mobile viewport         |
| Shared schema              | Unknown downstream consumers | Dependency not modeled | Unknown    | Run broader regression fallback |

The rows above are illustrative. Replace them with the current pull request.

Mark a dependency `Unknown` when the impact system or reviewer cannot bound it. Do not convert “no matched tests” into “no testing needed.”

## Step 3: Review test selection and assertions

For each affected behavior, require at least one of these dispositions:

* **Selected:** an existing test contains the required assertion.
* **Added:** a reviewed test or assertion was created for this change.
* **Covered elsewhere:** another control owns the risk, with a reference.
* **Accepted gap:** an authorized reviewer accepts the gap for this change.
* **Unknown:** the team cannot yet show coverage.

Use this worksheet:

| Behavior | Test reference | Required assertion | Disposition | Selection reason | Reviewer |
| -------- | -------------- | ------------------ | ----------- | ---------------- | -------- |
|          |                |                    |             |                  |          |

Check the assertion, not just the test name. A test called `admin can approve invoice` may only verify navigation. The gate needs the exact observable outcome that represents the risk.

### Agent-generated tests require review

An MCP-compatible coding agent can call ContextQA tools after the server is connected and authorized. Claude Code's official MCP guidance notes that connected servers can let the agent read and act on external systems, which is precisely why tool permissions and review boundaries matter. See [Connect Claude Code to tools via MCP](https://code.claude.com/docs/en/mcp).

Before counting an agent-generated test as coverage:

1. Review its preconditions and data dependencies.
2. Trace every acceptance criterion to a concrete assertion.
3. Run it once against the expected behavior.
4. Run it against a controlled failure when the risk warrants it.
5. Store the test revision or last-reviewed timestamp in the evidence packet.

## Step 4: Bind execution to the candidate build

The most dangerous green result is a correct test against the wrong build.

Record at least two independent identifiers when possible:

| Layer       | Identity to capture                                                 |
| ----------- | ------------------------------------------------------------------- |
| Source      | Full candidate commit SHA                                           |
| Artifact    | Image digest, package version, or immutable build ID                |
| Deployment  | Deployment ID or environment revision                               |
| Application | A version endpoint, footer build ID, or test-only metadata endpoint |
| Test        | Test-case or suite reference and revision                           |
| Run         | ContextQA execution or plan-run reference                           |

Add a preflight assertion that reads the application version when your architecture supports it. If the deployed version does not equal the candidate, return `Unknown: wrong build` before interpreting functional results.

### Wrong-build acceptance test

1. Deploy build A to the test environment.
2. Start a gate record for build B.
3. Run the preflight identity check.
4. Verify that the gate does not return `Pass`.
5. Confirm the output names both the expected and observed identities without exposing secrets.

## Step 5: Wait for a terminal result

ContextQA executions are asynchronous. Follow the documented [agent execution workflow](/mcp-server/agent-integration-guide.md) or the public start/status contract supplied to your workspace. Do not infer endpoint paths from portal traffic.

Use bounded polling:

```
Start the saved plan
  -> store the returned run reference
  -> poll at the documented interval
  -> stop on a documented terminal state
  -> stop at the gate timeout
  -> map success, failure, and incomplete states explicitly
```

The polling timeout is a gate outcome, not proof that the underlying test failed. Report `Unknown: gate timed out while run was <last observed state>` and keep the run reference for later review.

## Step 6: Assemble the evidence record

Use one record per pull-request evaluation.

```
PR:
Candidate commit:
Base commit:
Build/deployment identity:
Environment:

Affected behaviors:
Selected test references:
Required assertions:
Accepted gaps:
Unknown dependencies:

ContextQA run reference:
Test revision:
Started at:
Finished at:
Terminal state:
Gate outcome: Pass / Fail / Unknown

Failure evidence:
Missing or inaccessible evidence:
Reviewer:
Remediation owner and due date:
Final disposition:
```

Prefer a durable run reference and the [ContextQA test result](/reporting/test-results.md) over pasted screenshots alone. Screenshots are useful evidence, but they lose surrounding identity, step, and state context when detached.

## Step 7: Exercise the negative paths

Keep the gate advisory until this matrix passes.

| Test case                                 | Expected gate outcome       | What to verify                           |
| ----------------------------------------- | --------------------------- | ---------------------------------------- |
| All required assertions pass              | Pass                        | Correct build and evidence are linked    |
| Required assertion fails                  | Fail                        | Failure is visible and owned             |
| Run remains queued past gate timeout      | Unknown                     | Never becomes a pass                     |
| Run starts, then is canceled              | Unknown                     | Cancellation is preserved                |
| Required test is skipped                  | Unknown                     | Skip reason and owner are visible        |
| Candidate build does not match            | Unknown                     | Expected and observed identities appear  |
| Selection report is partial               | Unknown or broader fallback | Unprocessed scope is explicit            |
| No test matches a risky change            | Unknown or accepted gap     | Human decision is recorded               |
| Agent tool call is denied                 | Unknown                     | Permission failure is not a test failure |
| Evidence link is inaccessible to reviewer | Unknown                     | Access problem is actionable             |

Once the behavior is deterministic, use the [CI integration acceptance runbook](/evaluation-and-governance/ci-integration-acceptance-runbook.md) before making the check required.

## Assign ownership without hiding uncertainty

| Condition                   | Primary owner        | Required next action                           |
| --------------------------- | -------------------- | ---------------------------------------------- |
| Product assertion fails     | Feature team         | Diagnose and fix or document intended behavior |
| Test assertion is wrong     | Test owner           | Correct, review, and rerun                     |
| Coverage is missing         | QA/release owner     | Add coverage or approve the gap                |
| Environment is unavailable  | Platform owner       | Restore and rerun                              |
| Build identity mismatches   | Release engineering  | Correct deployment or metadata propagation     |
| Agent lacks tool permission | MCP/automation owner | Review least-privilege policy                  |
| Cause remains unclear       | Named investigator   | Time-box investigation; retain `Unknown`       |

An AI explanation may accelerate triage. It should not erase the named human owner or the evidence needed to close the decision.

## Review prompt for an MCP-compatible agent

Copy and adapt this prompt after connecting the agent to the approved ContextQA MCP server:

```
Evaluate pull request <PR URL> in advisory mode.

1. Record the full candidate and base commit SHAs.
2. Summarize changed paths and affected user behaviors.
3. Inspect the current PR Impact report if available; preserve Partial,
   Superseded, confidence, and unknown-dependency signals.
4. Propose existing tests for each behavior and quote the assertion intent.
5. Do not create, edit, approve, or run a test without the required permission.
6. Flag missing assertions and coverage as Unknown; do not infer a pass.
7. If execution is authorized, run only <saved plan reference>, record the
   returned run reference, and poll to a documented terminal state with a
   <timeout> limit.
8. Verify the application build identity equals <candidate identity>.
9. Return a Pass, Fail, or Unknown record using the fields in our gate template.
10. Never expose credentials, hidden test data, or raw authorization headers.
```

## Common questions

<details>

<summary>Should an AI agent be allowed to approve its own generated test?</summary>

Not by default. Separate creation from approval for release-blocking coverage. A qualified reviewer should verify assertion intent, data dependencies, and at least one representative execution.

</details>

<details>

<summary>What if every selected test passes but coverage is incomplete?</summary>

The selected tests passed; the release decision remains `Unknown` unless an authorized reviewer accepts the gap or a broader fallback run covers it.

</details>

<details>

<summary>Can the current PR Impact Analysis check block the build?</summary>

The documented initial rollout is advisory. Treat it as evidence for selection, then enforce a separate, tested CI gate only when the public execution contract and your acceptance behavior are ready.

</details>

## Related documentation

* [Evaluation and governance](/evaluation-and-governance/evaluation.md)
* [PR Impact Analysis](/ai-features/pr-impact-analysis.md)
* [Agent integration guide](/mcp-server/agent-integration-guide.md)
* [Prepare a CI/CD quality gate](/getting-started/ci-cd-tutorial.md)
* [API tokens](/administration/api-tokens.md)

{% hint style="info" %}
**Want to validate an agent-assisted quality gate on a real pull request?** [**Book a technical review →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=agentic_pr_evidence) — Bring one PR, its acceptance criteria, and the release check you use today.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/evaluation-and-governance/agentic-pr-evidence-checklist.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.