> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-features/pr-impact-analysis.md).
# PR Impact Analysis
Use ContextQA PR Impact Analysis to map a GitHub pull request to affected, reusable, or missing tests with confidence and code-change evidence.
## Quick answer
ContextQA PR Impact Analysis reads a GitHub pull request, maps its code changes to the test repository, and produces a reviewable report grouped by recommended action. Start in **Settings → PR Impact Analysis**, define the base branch and path scope, then open a **PR Impact** row in **Assets → Requirements**. Recommendations remain advisory and do not edit test cases or block the GitHub build; an authorized reviewer can manually run the impacted cases from an eligible completed analysis and open the resulting execution evidence.
## What this page covers
{% hint style="info" %}
**Who is this for?** QA engineers, SDETs, developers, and release owners who want change-based regression selection without relying on test names or tribal knowledge.
{% endhint %}
{% hint style="warning" %}
PR Impact Analysis is a limited-rollout capability. The settings route or PR Impact requirement type may be unavailable until ContextQA enables it for your workspace and connects an eligible GitHub repository.
{% endhint %}
## What PR Impact Analysis answers
For each pull request, the report answers three citation-friendly questions:
1. **Which tests are related to this change?** The report groups impacted tests by its recommended verdict.
2. **Why is each test included?** Expand a row to inspect code-change evidence, confidence, and the generated MCP prompt.
3. **How trustworthy is the recommendation?** The report identifies confidence and whether the match came from structural application mapping or semantic test knowledge.
This makes the analysis useful as both a release-review surface and an evidence trail for an AI agent preparing a regression run.
## Configure the analysis boundary
Open **Settings → PR Impact Analysis**.
The current settings page includes:
| Setting | What it controls |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Autonomy tier** | The operating mode. **Read** produces an advisory report and makes no test changes. Other tiers may be visible but are not part of the initial read-only rollout. |
| **Confidence threshold** | Recommendations below the threshold are held rather than treated as high-confidence guidance. |
| **Base branch filter** | Limits analysis to pull requests targeting the named branch, such as `main`. |
| **Max files per analysis** | Caps the files evaluated in one run. Reports beyond the limit are marked partial. |
| **Path scope** | One glob per line for the repository or monorepo subtree to include. |
| **Post PR comment** | Posts or updates the ContextQA summary on the pull request when enabled. |
| **Post GitHub check** | Adds an advisory GitHub check. The check does not fail the build. |
| **Suppress paths** | Excludes generated, vendor, or other noisy paths from analysis. |
{% hint style="info" %}
The settings snapshot is stored with an analysis so a reviewer can understand the boundary used at that time, even after an administrator changes the live configuration.
{% endhint %}
## How a pull request becomes a coverage report
```mermaid
flowchart LR
A[GitHub pull request updated] --> B{Matches base branch
and path scope?}
B -- No --> C[No analysis created]
B -- Yes --> D[ContextQA maps changed code
to test knowledge]
D --> E[PR Impact requirement]
E --> F[Overview: actions and confidence]
F --> G[Verdict tabs: impacted tests]
G --> H[Evidence, confidence, and MCP prompt]
H --> I[Reviewer selects the regression scope]
I --> J{Run impacted cases?}
J -- Yes --> K[Choose environment and start run]
K --> L[Watch live or open completed run]
J -- No --> M[Preserve analysis for review]
```
The GitHub comment is updated in place as the pull request changes, so reviewers do not need to follow a chain of obsolete comments. A newer commit can supersede an earlier analysis; ContextQA preserves the earlier report as history and links to the newer run.
## Review a PR Impact report
1. Open **Assets → Requirements**.
2. Find the row whose type is **PR Impact** and open its `REQ-` link.
3. Confirm the pull-request repository, target branch, commit range, author, and changed-file count in the header.
4. Read the **AI summary** and check for a **Partial** or **Superseded** notice.
5. Use **Overview** to review the recommended-action totals and average confidence.
6. Open each verdict tab to inspect the affected test cases.
7. Expand **Evidence, confidence & prompt** on a row.
8. Use the report to select or update the regression scope through your normal reviewed workflow.
9. When the analysis is eligible, select **Run impacted cases** to execute the recommended set, choose the intended environment, and confirm the run.
The expanded row can show:
* the linked test-case ID and title;
* priority and confidence;
* the reason the test was associated with the change;
* relevant code-change evidence;
* whether the signal is structural or semantic; and
* a generated MCP prompt that an authorized AI agent can use as a starting point.
{% hint style="warning" %}
The analysis is read-only with respect to test content: it does not approve, dismiss, edit, create, or retire a test by itself. Starting an impacted-case run is a separate, explicit reviewer action.
{% endhint %}
## Run the impacted cases
Use the manual run action after you have reviewed the selected tests and confirmed that the application build matches the analyzed pull-request commit.
1. Open an eligible completed PR Impact report.
2. Select **Run impacted cases** in the report header.
3. Review the number of cases and choose the correct execution environment.
4. Start the run.
5. While it is scheduled or running, select **Watch live** to follow execution.
6. When it completes, review the passed, failed, and not-run counts, then select **Open run** for step evidence.
{% hint style="warning" %}
An API or browser step with a hard-coded URL can still call an older deployment even when you select the intended ContextQA environment. Before treating the result as pull-request evidence, verify the build identifier and target URL used by the run.
{% endhint %}
A completed panel that reports zero executed tests is not a passing result. Treat it as an incomplete evidence set and investigate the selected cases, environment, and execution record. Failed or superseded analyses are not valid inputs for an impacted-case run; use the available full-suite fallback when the analysis cannot produce a safe scope.
## Read the result safely
* Treat **high confidence** as stronger evidence, not as proof that the test will pass.
* A **semantic** match means the diff aligns with test knowledge; a **structural** match means the changed file maps to an exercised route or component.
* A **partial** report did not analyze every changed file. Review the unprocessed scope before using it as a release gate.
* **No behavioural change detected** is a valid result. Confirm the repository and path settings before assuming no regression is possible.
* A failed analysis offers retry and full-suite fallback actions; it should not produce a false empty report.
## Measure selection quality with a run ledger
An impact report proposes a regression scope. To evaluate whether that scope is safe, compare it with a broader reference run on the same commit, environment, data revision, and test revision.
```mermaid
flowchart LR
A[Commit-specific impact report] --> B[Selected test run]
A --> C[Broader reference run]
B --> D[Compare failures and omissions]
C --> D
D --> E[Accept, tune, or fall back]
```
Use a time-bounded cohort of representative pull requests, including risky changes and changes that should not affect testable behavior. For each pull request, preserve the analysis settings snapshot and record:
| Field | What to capture |
| -------------------------------- | -------------------------------------------------------------------------- |
| Pull request and commit | Stable PR reference and full analyzed SHA |
| Analysis scope | Base branch, included paths, suppressed paths, and file cap |
| Report state | Complete, Partial, Superseded, Failed, or no behavioral change |
| Selected tests | Test-case references and selection reasons |
| Known relevant tests omitted | Tests a qualified reviewer expected but the report did not select |
| Selected tests judged irrelevant | Recommendations without a defensible relationship to the change |
| Unknown dependencies | Shared services, schemas, or runtime behavior the analysis could not bound |
| Fallback | Expanded set, broader suite, or explicit human disposition |
| Run evidence | Selected and reference run IDs, build, environment, and result |
| Reviewer | Person who classified the outcome |
### Use precise error definitions
For this evaluation:
* A **selection false negative** is a test judged relevant by the review protocol or a failure found in the broader reference run that the impact report did not select.
* A **selection false positive** is a selected test judged irrelevant under the same review protocol.
* An **unknown** is a dependency or outcome the reviewer cannot classify from the available evidence.
These are selection-quality labels, not claims that a product defect was or was not real. Write the review protocol before measuring so the same evidence is classified consistently.
### Copyable impact ledger
| Date | PR / SHA | Scope state | Selected count | Relevant omitted | Irrelevant selected | Unknown dependencies | Selected-run result | Reference-run result | Fallback used | Evidence / reviewer |
| ---- | -------- | ----------- | -------------: | ---------------: | ------------------: | -------------------: | ------------------- | -------------------- | ------------- | ------------------- |
| | | | | | | | | | | |
When a broader run finds a relevant failure outside the selected set, treat the event as a release-safety incident for the selection rule. Preserve the diff, analysis snapshot, omitted test, failure evidence, and remediation decision.
### Decide when to fall back
Use the broader regression scope when:
* the report is `Partial`, `Failed`, or based on a superseded commit;
* the changed file count exceeds the configured analysis boundary;
* a shared dependency is unknown;
* a critical workflow has no selected test and no accepted coverage disposition;
* the application build does not match the analyzed commit; or
* the cohort exceeds the team's accepted false-negative threshold.
Do not publish a universal precision or recall claim from a small internal sample. Report the cohort, scope, definitions, and unknowns with the result.
For the complete release-decision record, use the [agentic PR evidence checklist](/evaluation-and-governance/agentic-pr-evidence-checklist.md).
## FAQ
Does PR Impact Analysis change my test cases automatically?
No. It produces recommendations, evidence, and an MCP prompt while leaving the test repository unchanged. A reviewer can separately start a run of the impacted cases.
Does PR Impact Analysis run tests automatically?
The current documented workflow provides a manual **Run impacted cases** action for an eligible completed analysis. Automated pre-merge or post-deployment execution is not part of this public workflow.
Why does the run show zero executed tests?
Zero executed tests is not a pass. Open the run, verify the selected cases and environment, and use a broader regression scope if the impact set cannot produce valid execution evidence.
Does the GitHub check block a merge?
No. The current GitHub check is advisory and does not fail the build.
Why is a report marked partial?
The pull request exceeded the configured file limit or another analysis boundary. Review the report notice and expand the regression scope when needed.
## Related documentation
* [Requirements management](/web-testing/requirements-management.md)
* [Requirements coverage gaps](/ai-features/requirements-coverage-gaps.md)
* [GitHub Actions](/integrations/github-actions.md)
* [Running tests](/execution/running-tests.md)
* [MCP agent integration](/mcp-server/agent-integration-guide.md)
{% hint style="info" %}
**See AI change-impact analysis on your repository.** [**Book a Demo →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=pr_impact_analysis) — Review how ContextQA turns a pull request into an evidence-backed regression scope.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://learning.contextqa.com/ai-features/pr-impact-analysis.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.