> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/reporting/flaky-test-detection.md).

# Flaky Test Detection

## Quick answer

Flaky test detection in ContextQA — AI-powered failure classification identifies intermittent failures and separates them from real application bugs. Use this page to understand when the capability applies, complete its user-facing workflow, and verify the expected result.

## What this page covers

{% hint style="info" %}
**Who is this for?** QA managers and engineering managers who need to separate intermittent test noise from real regressions — so failures that matter get attention and flaky tests don't block deployments.
{% endhint %}

> **Flaky test:** A test case that produces inconsistent results — passing on some executions and failing on others — without any change to the application code or test definition, typically caused by timing issues, environment variability, or non-deterministic UI behavior.

Flaky tests are the primary reason development teams lose confidence in automated test suites. When every pipeline failure requires human triage to determine whether it is a real regression or noise, velocity drops and eventually the suite is ignored. ContextQA addresses this by classifying every failure with an AI-derived root cause category, making flakiness visible as a distinct failure type rather than an undifferentiated red status.

## What is a flaky test?

A flaky test passes on some runs and fails on others under conditions that have not changed — same code, same environment, same test definition. Common causes include:

* **Timing dependencies:** The test clicks a button before an async operation completes.
* **Order dependence:** The test relies on state left by a previous test case that sometimes runs in a different order.
* **Environment variability:** Network latency spikes, DNS resolution delays, or shared database contention.
* **Non-deterministic UI:** Animations, lazy-loaded components, or third-party widgets that render at unpredictable times.

ContextQA surfaces flakiness as a failure category distinct from an application bug. Treat a single-run AI classification as a triage hypothesis: confirm it with comparable execution history and the captured evidence. A real regression can also be intermittent, and a test problem can fail consistently.

## How ContextQA classifies failures

ContextQA uses AI root cause analysis on every failed test case. The analysis pipeline examines:

* The failing step and the error message
* The browser console log for JavaScript errors
* The HAR network log for failed or slow requests
* The DOM state at the time of failure (from the Playwright trace)
* Historical execution data for the same test case

From this data, ContextQA assigns one of four failure categories:

| Category              | Meaning                                                                                        |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| **Application Bug**   | The application behaved incorrectly; the test is functioning as designed                       |
| **Test Bug**          | The test definition has an error — incorrect selector, wrong expected value, missing wait      |
| **Flaky Failure**     | The failure is intermittent; root cause is timing, environment variability, or non-determinism |
| **Environment Issue** | Infrastructure-level failure — network timeout, missing credential, environment not responding |

The AI explanation accompanying each classification states specifically what evidence led to the classification. For a flaky failure, the explanation typically notes that the test has both passed and failed on identical code, and identifies the specific step and condition that is non-deterministic.

## Accessing flaky test data in the Analytics Dashboard

1. Open **Analytics** in the left navigation.
2. Click the **Execution Dashboard** tab.
3. Locate the **Consistently Failing Tests** widget. This widget lists test cases with repeated failures across recent runs, ranked by failure frequency.
4. Click any test case in the widget to open the failure detail panel.
5. The detail panel shows the failure category distribution (how many runs were classified as flaky vs. application bug vs. other) and the AI explanation for each failure type.

The **Consistently Failing Tests** widget surfaces test cases that failed in multiple consecutive runs. ContextQA distinguishes between cases that always fail (likely an application bug or test bug) and cases that alternate between passing and failing (likely flaky). The visual indicator for flakiness is a mixed pass/fail run history in the sparkline column.

For a broader view, the **Failure Analysis** report (accessed from **Analytics → Failure Analysis**) shows aggregate failure categories across an entire test suite or date range, allowing you to measure what percentage of your failures are flaky versus genuine regressions.

## Using the get\_root\_cause MCP tool

The `get_root_cause` MCP tool returns the AI failure classification for a specific execution programmatically. The response includes:

* `errorType`: one of `APPLICATION_BUG`, `TEST_BUG`, `FLAKY_FAILURE`, `ENVIRONMENT_ISSUE`
* `rootCause`: the AI's natural-language explanation of the root cause
* `fixSuggestion`: a concrete suggestion for resolving the failure
* `affectedStep`: the step number and action that failed

When building CI integrations that need to distinguish "block the release" from "likely noise," use the `errorType` field from `get_root_cause` to gate your pipeline logic. A pipeline that fails the build on `APPLICATION_BUG` but creates a Jira ticket and continues on `FLAKY_FAILURE` is a common pattern for teams managing large suites.

Example MCP invocation pattern:

```
get_root_cause(execution_id="<execution_id>")
→ { errorType: "FLAKY_FAILURE", rootCause: "...", fixSuggestion: "Add an explicit wait for the modal animation to complete before asserting." }
```

## What to do when a test is classified as flaky

ContextQA classifies the failure; resolving it requires one of three approaches depending on the root cause:

**1. Fix timing issues.** If the AI reasoning log identifies a race condition (for example, clicking an element before it is interactable), add an explicit wait step in the test case. In ContextQA's step editor, add a **Wait** action before the problematic step. The appropriate wait target is either a specific element becoming visible or a network request completing.

**2. Preserve the full plan result.** ContextQA test plans support a recovery action for failed test cases. In **Test Plans → \[Plan] → Settings**, `Run_Next_Testcase` records the failed case and continues with the remaining cases instead of halting the plan. This is continuation behavior, not a retry of the failed case. Use the completed plan result to compare failures without changing their evidence.

**3. Isolate environment dependencies.** If ContextQA consistently classifies failures as `ENVIRONMENT_ISSUE` for a specific test case, the test may be hitting a dependency that is unreliable in your staging environment. Use a dedicated test environment or mock the external dependency.

## Retry configuration and the recovery action

The **Recovery Action** in test plan settings controls what ContextQA does when a test case fails mid-plan:

| Recovery Action     | Behavior                                                    |
| ------------------- | ----------------------------------------------------------- |
| `Stop`              | Halt plan execution immediately on first failure            |
| `Run_Next_Testcase` | Mark the failed case and continue executing remaining cases |

`Run_Next_Testcase` lets a plan complete so you can review every case outcome. It does not rerun, convert, or suppress a failure. After the run, compare the failure category and evidence with prior executions before deciding whether to rerun anything.

## Frequently Asked Questions

### How many runs does ContextQA need before it can identify a flaky test?

Root-cause analysis can return a provisional `FLAKY_FAILURE` classification from one run when the available evidence suggests timing or environment variability. Confirm flakiness across comparable runs before changing release policy or suppressing an alert. Historical widgets require repeat executions to show a pattern.

### Does ContextQA automatically retry flaky tests?

ContextQA does not automatically retry an individual failed case within the same execution. `Run_Next_Testcase` continues the plan; it does not retry. If your team chooses to rerun a case or plan, preserve the original failure, define a small retry limit, and report both attempts so a passing retry cannot hide the first failure.

### Can I mark a test case as "known flaky" to suppress notifications?

There is no explicit "known flaky" flag on test cases in the current UI. The recommended approach is to use the failure category data from `get_root_cause` in your CI integration to suppress notifications for `FLAKY_FAILURE` classifications while still alerting on `APPLICATION_BUG`.

## Related

* [Failure analysis report](/reporting/failure-analysis.md)
* [Analytics dashboard](/reporting/analytics-dashboard.md)
* [Test results](/reporting/test-results.md)
* [Video recording and screenshots](/execution/video-and-screenshots.md)
* [Exporting reports](/reporting/exporting-reports.md)

{% hint style="info" %}
**Get release readiness reports your stakeholders understand.** [**Book a Demo →**](https://contextqa.com/book-a-demo/) — See the analytics dashboard, failure analysis, and flaky test detection for your test suite.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://learning.contextqa.com/reporting/flaky-test-detection.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.