> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/reporting/flaky-test-detection.md).

# Find and Fix Flaky Tests in CI

Diagnose flaky CI tests with repeatable Playwright runs, isolated state, and first-failure evidence, then use ContextQA to investigate managed test results.

## Quick answer

Preserve the first failure, reproduce it on the same build with controlled data, and compare timing, shared state, authentication, and environment differences. Repair the cause, then rerun under the original CI conditions. A passing retry or an AI “flaky” label is not proof that the product is correct. Use repository traces and ContextQA execution evidence to support the diagnosis, and record any coverage removed from a release gate.

## What this page covers

{% hint style="info" %}
**Who is this for?** Developers, QA engineers, and engineering managers investigating intermittent CI failures and deciding whether a repair restores meaningful release coverage.
{% endhint %}

> **Flaky test:** A test case that produces inconsistent results — passing on some executions and failing on others — without any change to the application code or test definition, typically caused by timing issues, environment variability, or non-deterministic UI behavior.

Repeated intermittent failures make a suite harder to trust. Investigate the behavior before treating a result as noise: an application race condition can be both intermittent and a real defect. ContextQA’s failure classification can help organize the investigation when analysis is available for the run.

## What is a flaky test?

A flaky test passes on some runs and fails on others under conditions that have not changed — same code, same environment, same test definition. Common causes include:

* **Timing dependencies:** The test clicks a button before an async operation completes.
* **Order dependence:** The test relies on state left by a previous test case that sometimes runs in a different order.
* **Environment variability:** Network latency spikes, DNS resolution delays, or shared database contention.
* **Non-deterministic UI:** Animations, lazy-loaded components, or third-party widgets that render at unpredictable times.

Treat any single-run AI diagnosis as a triage hypothesis: confirm it with comparable execution history and the captured evidence. A real regression can also be intermittent, and a test problem can fail consistently.

## Reproduce a flaky Playwright test in CI

Start with the original failing attempt. Save its application commit, test revision, runner and browser versions, project, worker count, role, fixture identifiers, and trace or logs. Preserve retries as separate attempts. Playwright reports a test that fails initially and passes on retry as flaky; that reporting category does not identify its cause. See [Playwright retries](https://playwright.dev/docs/test-retries).

For a disposable test environment and a known test file, this diagnostic command repeats a narrow case without retries:

```bash
npx playwright test tests/report-export.spec.ts --repeat-each=10 --retries=0 --workers=1 --trace retain-on-failure --reporter=line,html
```

Replace the path with your actual failing spec and retain the normal project and environment configuration. The example assumes Playwright is already installed in the repository; do not install or upgrade the runner during the same investigation. These options are described in the [Playwright CLI reference](https://playwright.dev/docs/test-cli).

Ten runs are a starting diagnostic sample, not a stability guarantee. One worker helps isolate concurrency effects, but a pass there does not resolve a failure that appears only with parallel workers. Compare the original CI worker count, browser projects, and test ordering after controlling disposable test data. Repetition can create many records; use isolated identities, unique identifiers, and the fixture’s normal cleanup.

## Match the evidence to a repair

| Evidence or hypothesis                        | What to check                                                                    | Candidate repair and verification                                                                                                                      |
| --------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Element visible before data is ready          | Trace, response timing, loading state, and expected value                        | Assert the observable ready state and value; do not merely add a fixed sleep                                                                           |
| Failures only under parallel workers          | Shared account, record IDs, cleanup, or test-order dependence                    | Give each test or worker its own state; rerun with the original parallel settings                                                                      |
| Session expires or belongs to the wrong role  | Server identity, cookie scope, expiry, and fixture reuse                         | Recreate the intended session and verify identity; use the [authentication guide](/testing-with-ai-coding-agents/playwright-authentication-testing.md) |
| Works locally but fails in CI                 | Browser version, resources, locale, timezone, fonts, base URL, and feature flags | Reproduce the meaningful differences; avoid blindly increasing timeouts                                                                                |
| Backend response is intermittently incorrect  | Request, response, server logs, and persisted state                              | Report and fix the application or dependency defect; a retry cannot make the behavior acceptable                                                       |
| Date or boundary assertion fails occasionally | Clock source, timezone, server/browser differences, and exact boundary           | Use a controlled clock where supported and test both sides of the requirement                                                                          |

Playwright’s [actionability checks](https://playwright.dev/docs/actionability) help determine whether an element can be acted on. They do not prove that your business transaction completed or that the server saved the value. Prefer web-first assertions for the relevant visible outcome and a persisted-state check where the requirement needs one.

## Ask an AI coding agent to investigate the first failure

```
Read the failing test, its fixture, and the first-attempt CI artifacts.
Keep the application and test revisions unchanged while diagnosing.
Separate observed facts from hypotheses. Propose one narrow reproduction
and identify any state it creates. Do not suppress errors, weaken assertions,
add blanket sleeps, or increase retries to make the check green.
After review, fix the supported cause and show the original assertion still
holds. Report first attempts, retries, skips, remaining uncertainty, and cleanup.
```

Change one relevant cause at a time. Verify the repaired test under the conditions that originally failed, then run affected neighboring tests. For a test-code repair, a controlled defect check in a disposable fixture can establish that the assertion still catches its intended failure. Keep before-and-after commands and artifacts with the change. See [testing AI-generated code](/testing-with-ai-coding-agents/testing-ai-generated-code.md) for a runnable example of that check.

## Investigate a managed ContextQA failure

[Connect ContextQA MCP](/mcp-server/installation-and-setup.md), inspect the authenticated tool manifest, and retrieve the result for the failed execution. Work from the specific **result ID**, so evidence from another attempt is not mistaken for the original failure.

1. Use `get_test_case_results` with the execution or result identifier accepted by its current schema. Check the actual status and verdict.
2. Use `get_test_step_results(result_id)` to identify the failed step and available step evidence.
3. Inspect `investigate_failure` and request the original evidence with `rerun` set to `false`.
4. Retrieve available console/network logs and trace links when needed. An empty or unavailable artifact must stay visible in the report; not every run has every evidence type.
5. Compare equivalent runs and the requirement before concluding that a failure is intermittent, a test defect, an application defect, or an environment problem.

Illustrative JSON arguments for `investigate_failure`—replace `1042` with the actual result ID returned for your run:

```json
{
  "result_id": 1042,
  "rerun": false
}
```

The tool returns the collected evidence. Use only the fields actually present; do not assume a fixed classification enum or a complete history of comparable runs. Its optional `rerun: true` can trigger a new test execution when a test-case ID is available. That is a separate side effect, not a prerequisite for reading the original evidence or proof that a new run has finished. See the [failure investigation reference](/mcp-server/tool-reference/support-and-migration.md#investigate_failure).

For portal-based investigation, open the failed case’s [execution history](/reporting/execution-history.md) and [test results](/reporting/test-results.md), then review the artifacts available for that run. The [analytics dashboard](/reporting/analytics-dashboard.md) can support trend review where the relevant data is available. Keep an AI diagnosis as a hypothesis until the evidence supports it.

Do not allow a release solely because an AI response describes the failure as flaky. Any temporary removal from a gate needs an owner, explicit risk acceptance, compensating coverage, and restoration criteria.

## What to do when a test is classified as flaky

Use the observed evidence to choose a repair; these are common investigation paths:

**1. Fix timing issues.** If the AI reasoning log identifies a race condition (for example, clicking an element before it is interactable), add an explicit wait step in the test case. In ContextQA's step editor, add a **Wait** action before the problematic step. The appropriate wait target is either a specific element becoming visible or a network request completing.

**2. Preserve the full plan result.** ContextQA test plans support a recovery action for failed test cases. In **Test Plans → \[Plan] → Settings**, `Run_Next_Testcase` records the failed case and continues with the remaining cases instead of halting the plan. This is continuation behavior, not a retry of the failed case. Use the completed plan result to compare failures without changing their evidence.

**3. Isolate environment dependencies.** If logs and comparable runs show an unreliable staging dependency, isolate that dependency before attributing the problem to the test. Use a dedicated test environment or mock the external dependency.

## Retry configuration and the recovery action

The **Recovery Action** in test plan settings controls what ContextQA does when a test case fails mid-plan:

| Recovery Action     | Behavior                                                    |
| ------------------- | ----------------------------------------------------------- |
| `Stop`              | Halt plan execution immediately on first failure            |
| `Run_Next_Testcase` | Mark the failed case and continue executing remaining cases |

`Run_Next_Testcase` lets a plan complete so you can review every case outcome. It does not rerun, convert, or suppress a failure. After the run, compare the failure category and evidence with prior executions before deciding whether to rerun anything.

## Assign flaky-test ownership and restoration criteria

Quarantine is a temporary risk decision, not a resolution. Track:

| Field                         | Example                                               |
| ----------------------------- | ----------------------------------------------------- |
| Test and affected requirement | Checkout tax total / REQ-142                          |
| First and latest failure      | Result IDs and dates                                  |
| Failure signature             | Timeout after tax request; screenshot and trace       |
| Reproduction rate             | 3 failures in 20 comparable runs                      |
| Coverage impact               | Release no longer proves displayed total              |
| Compensating control          | API total check until UI test is restored             |
| Owner and due date            | Payments QA / date                                    |
| Restoration gate              | 20 comparable clean runs plus root-cause fix evidence |

If a test is removed from a gate, name the business risk that is no longer covered and the person accepting it. Close the item only when the cause is fixed or the requirement is deliberately retired—not merely because recent reruns happened to pass.

## Frequently Asked Questions

### How many runs does ContextQA need before it can identify a flaky test?

There is no universal run count that establishes the cause. One result can suggest a timing or environment problem, but comparable executions are needed to demonstrate intermittency. Record the sample and conditions, and investigate the mechanism before changing release policy.

### Does ContextQA automatically retry flaky tests?

The documented `Run_Next_Testcase` recovery action continues the plan; it does not retry. The MCP investigation tool’s optional `rerun` parameter can request a separate execution. If your team chooses to rerun a case or plan, preserve the original failure, define a small retry limit, and report both attempts so a passing retry cannot hide the first failure.

### Can I mark a test case as "known flaky" to suppress notifications?

A dedicated “known flaky” flag is not documented here. Route repeated signatures to an owned investigation while preserving the original result. Do not suppress a new failure or bypass a release gate solely because an AI diagnosis calls it flaky; first establish comparable history, affected coverage, and an approved temporary handling rule.

### Is a passing retry enough to close a flaky-test bug?

No. Retain both attempts and explain the cause, repair, original reproduction conditions, and remaining uncertainty. A clean sample supports confidence only for the conditions exercised; it is not proof that the intermittent failure can never recur.

## Related

* [Playwright testing with AI agents](/testing-with-ai-coding-agents/playwright-testing.md)
* [Authentication and session testing](/testing-with-ai-coding-agents/playwright-authentication-testing.md)
* [Failure analysis report](/reporting/failure-analysis.md)
* [Analytics dashboard](/reporting/analytics-dashboard.md)
* [Test results](/reporting/test-results.md)
* [Video recording and screenshots](/execution/video-and-screenshots.md)
* [Exporting reports](/reporting/exporting-reports.md)

{% hint style="info" %}
**Get release readiness reports your stakeholders understand.** [**Book a Demo →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=flaky-tests-ci) — See the analytics dashboard, failure analysis, and flaky test detection for your test suite.
{% endhint %}

CI workflow and linked Playwright sources reviewed September 25, 2026. Adapt commands and restoration criteria to your repository and risk.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://learning.contextqa.com/reporting/flaky-test-detection.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.