Agentic PR Evidence Checklist

Define the intent-to-test evidence chain, terminal states, ownership, missing coverage, and wrong-build controls for an agent-assisted PR review.

Quick answer

An AI agent should not approve a pull request because it successfully called a test tool. A defensible gate must connect the change intent to affected paths, selected tests, reviewed assertions, the exact candidate build, a terminal execution result, and accessible evidence. Represent the outcome as Pass, Fail, or Unknown; keep the gate advisory until failure, timeout, cancellation, skipped-test, missing-coverage, and wrong-build cases behave as agreed.

What this guide produces

Use this checklist when Claude Code, another MCP-compatible agent, or CI automation helps select or run ContextQA tests for a pull request. It produces a review record that answers five questions:

1. What change is being evaluated?
2. Why do these tests cover it?
3. Which exact application and test versions ran?
4. Did the execution finish with a trustworthy result?
5. Who owns missing coverage or remediation?

This is a team-defined governance pattern. It does not imply that ContextQA automatically enforces the gate, edits tests, or maps every field below into its API. The current PR Impact Analysis experience is advisory and read-only.

Model the evidence chain

The gate is only as strong as its weakest link.

A broken link is not automatically a failure in the product. It is an Unknown release decision until a reviewer expands coverage, verifies identity, or reruns the test.

Define the three gate outcomes

Outcome	Meaning	CI behavior
Pass	Required assertions ran against the intended build and all blocking results passed	May allow the protected action after acceptance testing
Fail	At least one required assertion reached a terminal failing result	Block and assign a remediation owner
Unknown	The evidence is incomplete, mismatched, timed out, canceled, skipped unexpectedly, or missing required coverage	Block for a required gate; report neutrally for an advisory gate

Avoid a binary mapping that treats everything other than an explicit failure as success. Queued, running, canceled, timed out, partial, and no result are not passing test outcomes.

Step 1: Capture the change contract

Populate this record from the pull request and build system before asking an agent to select tests.

Field	Required value
Repository	Canonical repository identifier
Pull request	Number and stable URL
Candidate commit	Full commit SHA, not a moving branch name
Base commit	Merge base or comparison SHA
Build identifier	Immutable artifact, deployment, or image ID
Environment	Named target and revision
Change intent	One-paragraph author summary
Acceptance criteria	Reviewable behavioral outcomes
Risk labels	Auth, payments, data integrity, migration, permissions, or other relevant risks
Evaluation timestamp	When the evidence packet was assembled

If the pull request description conflicts with the diff, flag the mismatch. The change intent is an input, not proof.

Check repository and requirement freshness

Cached context becomes unsafe when it describes a different commit, requirement revision, generated artifact, dependency graph, or test mapping from the one under review. Record provenance before reusing prior analysis.

Context input	Required identity	Refresh trigger	Unknown behavior
Repository snapshot	Full commit SHA and included paths	Head/base changes, force-push, submodule or lockfile change	Do not select tests from stale snapshot
Requirements/acceptance criteria	Source URL/ID and revision timestamp	Requirement edit, reopened decision, conflicting source	Flag conflict; request owner decision
Dependency/context graph	Source inputs, build/repo revision, refresh time	Dependency, route, schema, or service map changes	Expand fallback coverage
Test inventory/mapping	Test IDs and last-reviewed revision	Test created, retired, changed, or moved	Requery authoritative test source
Environment/deployment	Immutable build/deployment ID	Redeploy, rollback, feature-flag change	Mark build evidence Unknown
Agent context pack	Version and source list	Any input above changes	Rebuild pack; do not patch memory selectively

Freshness record

Analysis ID:
Repository head/base SHAs:
Requirement IDs/revisions:
Dependency-graph revision and generated-at:
Test-inventory generated-at:
Environment/build identity:
Agent context-pack version:
Known stale or unresolved inputs:
Decision: Reuse / Refresh / Unknown with broader fallback
Reviewer:

The policy above is team-defined unless your connected ContextQA workflow exposes and enforces the same metadata. Do not claim that the product invalidates stale context automatically without a verified run.

Illustrative stale-versus-refreshed example

An earlier analysis at commit A selected checkout UI tests. Commit B adds a shared authorization-schema change after the cache was built. Reusing the commit-A mapping would omit a role-boundary test. A refreshed analysis adds the permission workflow; until that refresh finishes, the gate uses the broader regression fallback and reports Unknown dependency rather than No impact.

This example is illustrative. In a real review, preserve both reports and the actual selected-test difference.

Step 2: Map paths to affected behavior

Start with the ContextQA PR Impact Analysis report when it is available. Review its scope, confidence, and Partial or Superseded state. Then add human knowledge that is not visible in the code graph.

Changed path or dependency	Affected behavior	Signal	Confidence	Reviewer note
services/permissions/*	Role-based approval controls	Structural mapping	High	Check inherited role behavior
ui/invoice/*	Invoice approval workflow	Semantic match	Medium	Include mobile viewport
Shared schema	Unknown downstream consumers	Dependency not modeled	Unknown	Run broader regression fallback

The rows above are illustrative. Replace them with the current pull request.

Mark a dependency Unknown when the impact system or reviewer cannot bound it. Do not convert “no matched tests” into “no testing needed.”

Step 3: Review test selection and assertions

For each affected behavior, require at least one of these dispositions:

• Selected: an existing test contains the required assertion.
• Added: a reviewed test or assertion was created for this change.
• Covered elsewhere: another control owns the risk, with a reference.
• Accepted gap: an authorized reviewer accepts the gap for this change.
• Unknown: the team cannot yet show coverage.

Use this worksheet:

Behavior	Test reference	Required assertion	Disposition	Selection reason	Reviewer

Check the assertion, not just the test name. A test called admin can approve invoice may only verify navigation. The gate needs the exact observable outcome that represents the risk.

Agent-generated tests require review

An MCP-compatible coding agent can call ContextQA tools after the server is connected and authorized. Claude Code's official MCP guidance notes that connected servers can let the agent read and act on external systems, which is precisely why tool permissions and review boundaries matter. See Connect Claude Code to tools via MCP.

Before counting an agent-generated test as coverage:

1. Review its preconditions and data dependencies.
2. Trace every acceptance criterion to a concrete assertion.
3. Run it once against the expected behavior.
4. Run it against a controlled failure when the risk warrants it.
5. Store the test revision or last-reviewed timestamp in the evidence packet.

Step 4: Bind execution to the candidate build

The most dangerous green result is a correct test against the wrong build.

Record at least two independent identifiers when possible:

Layer	Identity to capture
Source	Full candidate commit SHA
Artifact	Image digest, package version, or immutable build ID
Deployment	Deployment ID or environment revision
Application	A version endpoint, footer build ID, or test-only metadata endpoint
Test	Test-case or suite reference and revision
Run	ContextQA execution or plan-run reference

Add a preflight assertion that reads the application version when your architecture supports it. If the deployed version does not equal the candidate, return Unknown: wrong build before interpreting functional results.

Wrong-build acceptance test

1. Deploy build A to the test environment.
2. Start a gate record for build B.
3. Run the preflight identity check.
4. Verify that the gate does not return Pass.
5. Confirm the output names both the expected and observed identities without exposing secrets.

Step 5: Wait for a terminal result

ContextQA executions are asynchronous. Follow the documented agent execution workflow or the public start/status contract supplied to your workspace. Do not infer endpoint paths from portal traffic.

Use bounded polling:

Start the saved plan
  -> store the returned run reference
  -> poll at the documented interval
  -> stop on a documented terminal state
  -> stop at the gate timeout
  -> map success, failure, and incomplete states explicitly

The polling timeout is a gate outcome, not proof that the underlying test failed. Report Unknown: gate timed out while run was <last observed state> and keep the run reference for later review.

Step 6: Assemble the evidence record

Use one record per pull-request evaluation.

PR:
Candidate commit:
Base commit:
Build/deployment identity:
Environment:

Affected behaviors:
Selected test references:
Required assertions:
Accepted gaps:
Unknown dependencies:

ContextQA run reference:
Test revision:
Started at:
Finished at:
Terminal state:
Gate outcome: Pass / Fail / Unknown

Failure evidence:
Missing or inaccessible evidence:
Reviewer:
Remediation owner and due date:
Final disposition:

Prefer a durable run reference and the ContextQA test result over pasted screenshots alone. Screenshots are useful evidence, but they lose surrounding identity, step, and state context when detached.

Step 7: Exercise the negative paths

Keep the gate advisory until this matrix passes.

Test case	Expected gate outcome	What to verify
All required assertions pass	Pass	Correct build and evidence are linked
Required assertion fails	Fail	Failure is visible and owned
Run remains queued past gate timeout	Unknown	Never becomes a pass
Run starts, then is canceled	Unknown	Cancellation is preserved
Required test is skipped	Unknown	Skip reason and owner are visible
Candidate build does not match	Unknown	Expected and observed identities appear
Selection report is partial	Unknown or broader fallback	Unprocessed scope is explicit
No test matches a risky change	Unknown or accepted gap	Human decision is recorded
Agent tool call is denied	Unknown	Permission failure is not a test failure
Evidence link is inaccessible to reviewer	Unknown	Access problem is actionable

Once the behavior is deterministic, use the CI integration acceptance runbook before making the check required.

Assign ownership without hiding uncertainty

Condition	Primary owner	Required next action
Product assertion fails	Feature team	Diagnose and fix or document intended behavior
Test assertion is wrong	Test owner	Correct, review, and rerun
Coverage is missing	QA/release owner	Add coverage or approve the gap
Environment is unavailable	Platform owner	Restore and rerun
Build identity mismatches	Release engineering	Correct deployment or metadata propagation
Agent lacks tool permission	MCP/automation owner	Review least-privilege policy
Cause remains unclear	Named investigator	Time-box investigation; retain Unknown

An AI explanation may accelerate triage. It should not erase the named human owner or the evidence needed to close the decision.

Review prompt for an MCP-compatible agent

Copy and adapt this prompt after connecting the agent to the approved ContextQA MCP server:

Evaluate pull request <PR URL> in advisory mode.

1. Record the full candidate and base commit SHAs.
2. Summarize changed paths and affected user behaviors.
3. Inspect the current PR Impact report if available; preserve Partial,
   Superseded, confidence, and unknown-dependency signals.
4. Propose existing tests for each behavior and quote the assertion intent.
5. Do not create, edit, approve, or run a test without the required permission.
6. Flag missing assertions and coverage as Unknown; do not infer a pass.
7. If execution is authorized, run only <saved plan reference>, record the
   returned run reference, and poll to a documented terminal state with a
   <timeout> limit.
8. Verify the application build identity equals <candidate identity>.
9. Return a Pass, Fail, or Unknown record using the fields in our gate template.
10. Never expose credentials, hidden test data, or raw authorization headers.

Common questions

Should an AI agent be allowed to approve its own generated test?

Not by default. Separate creation from approval for release-blocking coverage. A qualified reviewer should verify assertion intent, data dependencies, and at least one representative execution.

What if every selected test passes but coverage is incomplete?

The selected tests passed; the release decision remains Unknown unless an authorized reviewer accepts the gap or a broader fallback run covers it.

Can the current PR Impact Analysis check block the build?

The documented initial rollout is advisory. Treat it as evidence for selection, then enforce a separate, tested CI gate only when the public execution contract and your acceptance behavior are ready.

Related documentation

• Evaluation and governance
• PR Impact Analysis
• Agent integration guide
• Prepare a CI/CD quality gate
• API tokens

Want to validate an agent-assisted quality gate on a real pull request? Book a technical review → — Bring one PR, its acceptance criteria, and the release check you use today.