> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/reporting/failure-analysis.md).

# Failure Analysis

{% hint style="info" %}
**Who is this for?** QA managers and engineering managers who need to quickly diagnose test failures, distinguish real bugs from flaky tests, and push defects directly to Jira or Azure DevOps.
{% endhint %}

When a test fails, the goal is to determine as fast as possible whether the failure represents a real bug in the application, a flaky environmental issue, or a test that needs updating. ContextQA provides several tools to answer that question without manual log analysis.

***

## Accessing failure analysis from the results page

After a test execution completes:

1. Open the execution result page (via **View Detailed Report** in the post-run pop-up, or via **Execution Dashboard** → run history).
2. Click the failed step or the failing test case card.
3. The failure detail panel expands showing the error message, screenshot, and network log entries captured at the moment of failure.

From this panel you can access the full **Root Cause Analysis** view by clicking the root cause indicator or navigating to the failure analysis section.

***

## Root cause analysis

ContextQA's AI engine analyzes failed steps and produces a structured root cause report. The report contains:

* **Failure category** — a classification of the failure type (e.g., element not found, assertion mismatch, network error, timeout)
* **AI explanation** — a plain-English description of what went wrong and why, generated from the screenshot, DOM state, and error message captured at the point of failure
* **Fix suggestion** — a recommended action to resolve the failure, such as updating a locator, adding a wait step, or investigating a specific API endpoint

The AI explanation and fix suggestion are displayed directly in the result page so a developer or QA engineer can act on them without needing to reproduce the failure locally first.

### Root cause categories

The AI classifies each failure into one of the following categories. The category tells you where the problem lies and who should act on it:

| Category                         | What it means                                                                                                              |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Script Failure**               | The automation script failed to execute as written — a locator, wait, or step in the test broke. Fix the step and re-run.  |
| **Functional / Product Bug**     | The application under test behaved incorrectly. A genuine product defect — report it to the development team.              |
| **Test Data Issue**              | Invalid, missing, or stale test data caused the failure. Update the test data and re-run.                                  |
| **Environment / Infrastructure** | A network, server, or deployment problem caused the failure. Check the environment and retry.                              |
| **ContextQA Platform**           | The failure originated in the ContextQA platform itself. Contact ContextQA support if it persists.                         |
| **Flaky / Intermittent**         | An unstable, non-deterministic failure that doesn't reproduce consistently. Add waits or retries to stabilize the step.    |
| **Configuration / Setup**        | A misconfiguration or incomplete setup caused the failure. Review the test or application configuration and prerequisites. |
| **Dependency / Integration**     | A failure in an external dependency or integration. Verify the dependent API or third-party service.                       |
| **Unclassified / Needs Review**  | The failure couldn't be automatically categorized and requires manual review.                                              |

### Debug mode for interactive diagnosis

ContextQA includes a **Debug** mode that lets you diagnose failures interactively as a test runs. When a step fails during a debug session, you are presented with options to handle it without stopping the run:

* **Enter more detail** — provide a plain-English instruction to clarify the step (e.g., "click on admin")
* **Update and Rerun** — modify the failing step and retry from that point
* **Add a New Step and Rerun** — insert a new step at the failure point and resume
* **Skip Step** — bypass the failed step and continue with the remaining steps
* **Delete Step** — remove the step from the test case entirely

This workflow is useful during test authoring and maintenance when you want to iterate on a test case in real time rather than running it repeatedly from scratch.

***

## Suggested fixes

{% hint style="info" %}
**Availability:** Suggested fixes require the **auto-fix** capability on your plan. When it's not enabled, the root cause report still shows the failure category and AI explanation, but the apply-fix actions described below don't appear. Contact your account team to enable it.
{% endhint %}

When the auto-fix capability is enabled, ContextQA does more than explain a failure — it proposes concrete changes to the test and lets you apply them without opening the step editor. This turns root cause analysis into a review-and-apply repair workflow for script-level failures.

Unlike [AI self-healing](#ai-self-healing-and-auto-healed-steps), which repairs locator drift automatically while a test runs, suggested fixes are proposed after a run fails and applied by you after review. A suggested fix can update a step, add a step, or remove a step — not only change a locator — and always requires a re-run to confirm the fix works.

### Opening the suggested fixes panel

On the execution result page for a failed run:

1. Select the **Failure analysis & suggested fixes** button at the top of the report. After a fix has been applied, this button is replaced by a green **Fix applied · See fix** button (or **Auto-fixed · See fix** when the fix was applied automatically) that reopens the same panel.
2. The **AI Debug** tab opens and scrolls to the **Root Cause Analysis** panel, which shows the failure category, the AI explanation, and the proposed fix for each affected step.

### Types of proposed change

Each proposed fix targets one step and falls into one of three change types:

| Change type   | Panel label         | What it does                                                                                                                                                    |
| ------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Update a step | **Proposed change** | Modifies an existing step — for example, replaces a broken locator with a **Proposed locator** plus one or more **Fallback** locators, or updates a data value. |
| Add a step    | **New step**        | Inserts a new step at the failure point.                                                                                                                        |
| Remove a step | **Step removal**    | Deletes a step that is no longer valid.                                                                                                                         |

### Applying a fix

* **Apply fix** — applies the proposed change for a single step. Use this to review and apply changes one at a time.
* **Apply all fixes** — applies every not-yet-applied step in one operation. Steps that are already applied are skipped.

After a fix is applied, ContextQA confirms with **Step updated. Re-run the test to verify.** (or **Steps updated. Re-run the test to verify.** for multiple steps). The applied steps are marked as fixed and their **Apply fix** control is hidden. When every proposed step has been applied, the run is marked as fixed and the top-of-report button switches to **Fix applied · See fix**.

{% hint style="warning" %}
Applying a fix changes your test case immediately. Re-run the test afterward to confirm the change resolves the failure — applying a fix doesn't re-execute the test for you.
{% endhint %}

### Other actions in the panel

Beyond applying fixes, the panel lets you:

* **Jump to failed step** — scrolls to and highlights the failing step in the step list.
* **Re-run** — re-executes the test case to verify a fix.
* **Edit manually** — opens the test case in the [Test Steps Editor](/web-testing/test-steps-editor.md) so you can make the change yourself instead of applying the proposal.
* **Mark as bug** — creates a Jira bug from the failure when the root cause is a genuine product defect rather than a test issue. This requires a connected Jira integration; if none is configured, ContextQA prompts you to connect Jira in **Settings → Integrations**. See [Jira Integration](/integrations/jira.md).

***

## AI self-healing and auto-healed steps

When the application UI changes in a way that breaks a locator, ContextQA's AI self-healing engine attempts to automatically identify the correct new element and update the step. If the accuracy score is above 90% and there is no functional change to the flow, the step is healed automatically.

Healed steps are marked with an **auto-healed** indicator in the execution report. This means the test passed using an updated locator — not the original one — so you should review the healed step and confirm the update is correct before relying on it long-term.

To approve or review auto-healing suggestions, use the `approve_auto_healing` MCP tool or navigate to the auto-healing review queue in the platform.

***

## Flaky test detection

A test is considered flaky when it produces inconsistent results across runs on the same code and the same environment. ContextQA tracks pass/fail history per test case and surfaces tests that show non-deterministic behavior.

Flaky tests appear in the **Consistently Failing** list on the Execution Dashboard, often alongside a root cause note indicating intermittent failures. Use the failure trend data — available on the test case detail view — to see how often a given test has failed over recent runs and whether the failures cluster around specific times, environments, or deployments.

Common flaky failure categories surfaced by ContextQA:

* **Timing issues** — a step executing before a page element has fully loaded
* **Network variability** — a step depending on an API response that is occasionally slow
* **Data state issues** — test data that is not reset between runs

***

## Failure trends

The **Consistently Failing** panel in the Execution Dashboard lists test cases ranked by failure frequency. Each card shows the test case name, the failure count, and the AI-identified root cause when available.

Use this list to prioritize maintenance work. Test cases that appear repeatedly with the same root cause are strong candidates for a locator update or a stability fix. Test cases with varied root causes may indicate genuine application instability rather than test fragility.

***

## Integrating failures with bug trackers

Once you identify a genuine application bug from a failing test, you can push it directly to your bug tracker without leaving ContextQA.

### Reporting to Jira

From the test result detail page:

1. Click the **JIRA** option in the failure reporting toolbar.
2. Select your Jira project and set the issue type to **Bug**.
3. Click **Save** — a Jira ticket is created and linked to the ContextQA result.

The Jira issue is pre-populated with the test description, reproduction steps, and the result page link. See [Jira Integration](/integrations/jira.md) for setup instructions.

You can also create defect tickets programmatically using the `create_defect_ticket` MCP tool:

```
create_defect_ticket(
  execution_id="<execution_id>",
  project_id="<your-project>"
)
```

### Reporting to Azure DevOps

From the test result detail page:

1. Click **Report Bug** in the failure reporting toolbar.
2. Select **Azure Board** as the destination.
3. Choose the issue type (e.g., Bug, Test Case) and enter a title.
4. Click **Save** — ContextQA automatically adds the test description and reproduction steps to the work item.

A direct link to the Azure Boards ticket appears immediately after creation. See [Azure DevOps Integration](/integrations/azure-devops.md) for setup instructions.

{% hint style="info" %}
**Get release readiness reports your stakeholders understand.** [**Book a Demo →**](https://contextqa.com/book-a-demo/) — See the analytics dashboard, failure analysis, and flaky test detection for your test suite.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://learning.contextqa.com/reporting/failure-analysis.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.