> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-features/pr-impact-analysis.md).
# PR Impact Analysis
Use ContextQA PR Impact Analysis to map a GitHub pull request to affected, reusable, or missing tests with confidence and code-change evidence.
## Quick answer
ContextQA PR Impact Analysis maps a GitHub or GitLab change to affected tests and produces a reviewable report. In **Settings → PR Impact Analysis**, choose whether ContextQA should report only, run after merge, or run before merge; select the execution environment and deployment timing. The report also checks whether fixed URLs or missing environment variables would prevent the impacted cases from testing the intended build.
## What this page covers
{% hint style="info" %}
**Who is this for?** QA engineers, SDETs, developers, and release owners who want change-based regression selection without relying on test names or tribal knowledge.
{% endhint %}
{% hint style="warning" %}
PR Impact Analysis is a limited-rollout capability. The settings page or PR Impact requirement type may be unavailable until ContextQA enables it for your workspace and connects an eligible repository.
{% endhint %}
## What PR Impact Analysis answers
For each eligible pull or merge request, the report answers three citation-friendly questions:
1. **Which tests are related to this change?** The report groups impacted tests by its recommended verdict.
2. **Why is each test included?** Expand a row to inspect code-change evidence, confidence, and the generated MCP prompt.
3. **How trustworthy is the recommendation?** The report identifies confidence and whether the match came from structural application mapping or semantic test knowledge.
This makes the analysis useful as both a release-review surface and an evidence trail for an AI agent preparing a regression run.
## Configure the analysis boundary
Open **Settings → PR Impact Analysis**.
The current settings page includes:
| Setting | What it controls |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Autonomy tier** | The operating mode. **Read** produces an advisory report and makes no test changes. Other tiers may be visible but are not part of the initial read-only rollout. |
| **Confidence threshold** | Recommendations below the threshold are held rather than treated as high-confidence guidance. |
| **Base branch filter** | Limits analysis to pull requests targeting the named branch, such as `main`. |
| **Max files per analysis** | Caps the files evaluated in one run. Reports beyond the limit are marked partial. |
| **Path scope** | One glob per line for the repository or monorepo subtree to include. |
| **When should we run your tests?** | Report without executing, run after merge, or run against the pull request build before merge. |
| **Environment** | Supplies the variables and destination for automatic and manual impacted-case runs. |
| **Wait before running** | Gives a merged build time to deploy before an automatic run starts. |
| **Deployment environment name** | Optionally waits for the provider's named deployment when the same commit is deployed more than once. |
| **Post PR comment** | Posts or updates the ContextQA summary on the pull request when enabled. |
| **Post status check** | Adds an advisory GitHub check or GitLab commit status. It does not fail the build. |
| **Suppress paths** | Excludes generated, vendor, or other noisy paths from analysis. |
{% hint style="info" %}
The settings snapshot is stored with an analysis so a reviewer can understand the boundary used at that time, even after an administrator changes the live configuration.
{% endhint %}
## How a pull request becomes a coverage report
```mermaid
flowchart LR
A[GitHub pull request updated] --> B{Matches base branch
and path scope?}
B -- No --> C[No analysis created]
B -- Yes --> D[ContextQA maps changed code
to test knowledge]
D --> E[PR Impact requirement]
E --> F[Overview: actions and confidence]
F --> G[Verdict tabs: impacted tests]
G --> H[Evidence, confidence, and MCP prompt]
H --> I[Reviewer selects the regression scope]
I --> J{Configured timing}
J -- Report only --> K[Reviewer can start a manual run]
J -- After merge --> L[Wait for the intended deployment]
J -- Before merge --> M[Run against the pull request build]
K --> N[Open live or completed run]
L --> N
M --> N
```
When provider reporting is enabled, the pull- or merge-request summary is updated in place as the change evolves, so reviewers do not need to follow a chain of obsolete updates. A newer commit can supersede an earlier analysis; ContextQA preserves the earlier report as history and links to the newer run.
## Review a PR Impact report
1. Open **Assets → Requirements**.
2. Find the row whose type is **PR Impact** and open its `REQ-` link.
3. Confirm the pull-request repository, target branch, commit range, author, and changed-file count in the header.
4. Use **View pull request** when you need to compare the ContextQA report with the current provider record.
5. Read the **AI summary** and check for a **Partial** or **Superseded** notice.
6. Use **Overview** to review the recommended-action totals and average confidence.
7. Open each verdict tab to inspect the affected test cases. A pull-request link on an eligible row opens the provider record without changing the report decision.
8. Expand **Evidence, confidence & prompt** on a row.
9. Use the report to select or update the regression scope through your normal reviewed workflow.
10. When the analysis is eligible, select **Run impacted cases** to execute the recommended set with the environment saved in PR Impact settings.
Before using the result as release evidence, read the preflight panel directly below the run state. A clean panel says every checked case takes its address from the selected environment. A warning identifies cases that can test the wrong destination or fail before the product change is exercised, including:
* a hard-coded address;
* an environment variable that the selected environment does not define;
* an address embedded in AI step text;
* an address passed into a step group; or
* a global variable used as the destination.
The expanded row can show:
* the linked test-case ID and title;
* priority and confidence;
* the reason the test was associated with the change;
* relevant code-change evidence;
* whether the signal is structural or semantic; and
* a generated MCP prompt that an authorized AI agent can use as a starting point.
{% hint style="warning" %}
The analysis is read-only with respect to test content: it does not approve, dismiss, edit, create, or retire a test by itself. Starting an impacted-case run is a separate, explicit reviewer action.
{% endhint %}
## Run the impacted cases
Use the manual run action after you have reviewed the selected tests and confirmed that the application build matches the analyzed pull-request commit. The report uses the environment configured in PR Impact settings, and the completed run displays that environment with its evidence.
1. Open an eligible completed PR Impact report.
2. Select **Run impacted cases** in the report header.
3. Review the number of cases and confirm that PR Impact settings point to the correct execution environment.
4. Start the run from the report.
5. While it is scheduled or running, select **Watch live** to follow execution.
6. When it completes, review the passed, failed, and not-run counts, then select **Open run** for step evidence.
{% hint style="warning" %}
An API or browser step with a hard-coded URL can still call an older deployment even when you select the intended ContextQA environment. Before treating the result as pull-request evidence, verify the build identifier and target URL used by the run.
{% endhint %}
A completed panel that reports zero executed tests is not a passing result. Treat it as an incomplete evidence set and investigate the selected cases, environment, and execution record. Failed or superseded analyses are not valid inputs for an impacted-case run; use the available full-suite fallback when the analysis cannot produce a safe scope.
## Read the result safely
* Treat **high confidence** as stronger evidence, not as proof that the test will pass.
* A **semantic** match means the diff aligns with test knowledge; a **structural** match means the changed file maps to an exercised route or component.
* A **partial** report did not analyze every changed file. Review the unprocessed scope before using it as a release gate.
* **No behavioural change detected** is a valid result. Confirm the repository and path settings before assuming no regression is possible.
* A failed analysis offers retry and full-suite fallback actions; it should not produce a false empty report.
## Measure selection quality with a run ledger
An impact report proposes a regression scope. To evaluate whether that scope is safe, compare it with a broader reference run on the same commit, environment, data revision, and test revision.
```mermaid
flowchart LR
A[Commit-specific impact report] --> B[Selected test run]
A --> C[Broader reference run]
B --> D[Compare failures and omissions]
C --> D
D --> E[Accept, tune, or fall back]
```
Use a time-bounded cohort of representative pull requests, including risky changes and changes that should not affect testable behavior. For each pull request, preserve the analysis settings snapshot and record:
| Field | What to capture |
| -------------------------------- | -------------------------------------------------------------------------- |
| Pull request and commit | Stable PR reference and full analyzed SHA |
| Analysis scope | Base branch, included paths, suppressed paths, and file cap |
| Report state | Complete, Partial, Superseded, Failed, or no behavioral change |
| Selected tests | Test-case references and selection reasons |
| Known relevant tests omitted | Tests a qualified reviewer expected but the report did not select |
| Selected tests judged irrelevant | Recommendations without a defensible relationship to the change |
| Unknown dependencies | Shared services, schemas, or runtime behavior the analysis could not bound |
| Fallback | Expanded set, broader suite, or explicit human disposition |
| Run evidence | Selected and reference run IDs, build, environment, and result |
| Reviewer | Person who classified the outcome |
### Use precise error definitions
For this evaluation:
* A **selection false negative** is a test judged relevant by the review protocol or a failure found in the broader reference run that the impact report did not select.
* A **selection false positive** is a selected test judged irrelevant under the same review protocol.
* An **unknown** is a dependency or outcome the reviewer cannot classify from the available evidence.
These are selection-quality labels, not claims that a product defect was or was not real. Write the review protocol before measuring so the same evidence is classified consistently.
### Copyable impact ledger
| Date | PR / SHA | Scope state | Selected count | Relevant omitted | Irrelevant selected | Unknown dependencies | Selected-run result | Reference-run result | Fallback used | Evidence / reviewer |
| ---- | -------- | ----------- | -------------: | ---------------: | ------------------: | -------------------: | ------------------- | -------------------- | ------------- | ------------------- |
| | | | | | | | | | | |
When a broader run finds a relevant failure outside the selected set, treat the event as a release-safety incident for the selection rule. Preserve the diff, analysis snapshot, omitted test, failure evidence, and remediation decision.
### Decide when to fall back
Use the broader regression scope when:
* the report is `Partial`, `Failed`, or based on a superseded commit;
* the changed file count exceeds the configured analysis boundary;
* a shared dependency is unknown;
* a critical workflow has no selected test and no accepted coverage disposition;
* the application build does not match the analyzed commit; or
* the cohort exceeds the team's accepted false-negative threshold.
Do not publish a universal precision or recall claim from a small internal sample. Report the cohort, scope, definitions, and unknowns with the result.
For the complete release-decision record, use the [agentic PR evidence checklist](/evaluation-and-governance/agentic-pr-evidence-checklist.md).
## FAQ
Does PR Impact Analysis change my test cases automatically?
No. It produces recommendations, evidence, and an MCP prompt while leaving the test repository unchanged. A reviewer can separately start a run of the impacted cases.
Can PR Impact Analysis run tests automatically?
Yes, when an administrator selects **Before it merges** or **After the pull request merges** in PR Impact settings. Select **Don't run them** for analysis-only operation. An eligible report can still expose **Run impacted cases** as an explicit reviewer action.
Why does the run show zero executed tests?
Zero executed tests is not a pass. Open the run, verify the selected cases and environment, and use a broader regression scope if the impact set cannot produce valid execution evidence.
Does the provider status check block a merge?
No. The current GitHub check or GitLab commit status is advisory and does not fail the build.
Why is a report marked partial?
The pull request exceeded the configured file limit or another analysis boundary. Review the report notice and expand the regression scope when needed.
## Related documentation
* [Requirements management](/web-testing/requirements-management.md)
* [Requirements coverage gaps](/ai-features/requirements-coverage-gaps.md)
* [GitHub Actions](/integrations/github-actions.md)
* [Running tests](/execution/running-tests.md)
* [MCP agent integration](/mcp-server/agent-integration-guide.md)
{% hint style="info" %}
**See AI change-impact analysis on your repository.** [**Book a Demo →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=pr_impact_analysis) — Review how ContextQA turns a pull request into an evidence-backed regression scope.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://learning.contextqa.com/ai-features/pr-impact-analysis.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.