Failure Analysis
How to use ContextQA's failure analysis, root cause, flaky test detection, and bug tracker integration features to diagnose and resolve test failures quickly.
Who is this for? QA managers and engineering managers who need to quickly diagnose test failures, distinguish real bugs from flaky tests, and push defects directly to Jira or Azure DevOps.
When a test fails, the goal is to determine as fast as possible whether the failure represents a real bug in the application, a flaky environmental issue, or a test that needs updating. ContextQA provides several tools to answer that question without manual log analysis.
Accessing failure analysis from the results page
After a test execution completes:
Open the execution result page (via View Detailed Report in the post-run pop-up, or via Execution Dashboard → run history).
Click the failed step or the failing test case card.
The failure detail panel expands showing the error message, screenshot, and network log entries captured at the moment of failure.
From this panel you can access the full Root Cause Analysis view by clicking the root cause indicator or navigating to the failure analysis section.
Root cause analysis
ContextQA's AI engine analyzes failed steps and produces a structured root cause report. The report contains:
Failure category — a classification of the failure type (e.g., element not found, assertion mismatch, network error, timeout)
AI explanation — a plain-English description of what went wrong and why, generated from the screenshot, DOM state, and error message captured at the point of failure
Fix suggestion — a recommended action to resolve the failure, such as updating a locator, adding a wait step, or investigating a specific API endpoint
The AI explanation and fix suggestion are displayed directly in the result page so a developer or QA engineer can act on them without needing to reproduce the failure locally first.
Debug mode for interactive diagnosis
ContextQA includes a Debug mode that lets you diagnose failures interactively as a test runs. When a step fails during a debug session, you are presented with options to handle it without stopping the run:
Enter more detail — provide a plain-English instruction to clarify the step (e.g., "click on admin")
Update and Rerun — modify the failing step and retry from that point
Add a New Step and Rerun — insert a new step at the failure point and resume
Skip Step — bypass the failed step and continue with the remaining steps
Delete Step — remove the step from the test case entirely
This workflow is useful during test authoring and maintenance when you want to iterate on a test case in real time rather than running it repeatedly from scratch.
AI self-healing and auto-healed steps
When the application UI changes in a way that breaks a locator, ContextQA's AI self-healing engine attempts to automatically identify the correct new element and update the step. If the accuracy score is above 90% and there is no functional change to the flow, the step is healed automatically.
Healed steps are marked with an auto-healed indicator in the execution report. This means the test passed using an updated locator — not the original one — so you should review the healed step and confirm the update is correct before relying on it long-term.
To approve or review auto-healing suggestions, use the approve_auto_healing MCP tool or navigate to the auto-healing review queue in the platform.
Flaky test detection
A test is considered flaky when it produces inconsistent results across runs on the same code and the same environment. ContextQA tracks pass/fail history per test case and surfaces tests that show non-deterministic behavior.
Flaky tests appear in the Consistently Failing list on the Execution Dashboard, often alongside a root cause note indicating intermittent failures. Use the failure trend data — available on the test case detail view — to see how often a given test has failed over recent runs and whether the failures cluster around specific times, environments, or deployments.
Common flaky failure categories surfaced by ContextQA:
Timing issues — a step executing before a page element has fully loaded
Network variability — a step depending on an API response that is occasionally slow
Data state issues — test data that is not reset between runs
Failure trends
The Consistently Failing panel in the Execution Dashboard lists test cases ranked by failure frequency. Each card shows the test case name, the failure count, and the AI-identified root cause when available.
Use this list to prioritize maintenance work. Test cases that appear repeatedly with the same root cause are strong candidates for a locator update or a stability fix. Test cases with varied root causes may indicate genuine application instability rather than test fragility.
Integrating failures with bug trackers
Once you identify a genuine application bug from a failing test, you can push it directly to your bug tracker without leaving ContextQA.
Reporting to Jira
From the test result detail page:
Click the JIRA option in the failure reporting toolbar.
Select your Jira project and set the issue type to Bug.
Click Save — a Jira ticket is created and linked to the ContextQA result.
The Jira issue is pre-populated with the test description, reproduction steps, and the result page link. See Jira Integration for setup instructions.
You can also create defect tickets programmatically using the create_defect_ticket MCP tool:
Reporting to Azure DevOps
From the test result detail page:
Click Report Bug in the failure reporting toolbar.
Select Azure Board as the destination.
Choose the issue type (e.g., Bug, Test Case) and enter a title.
Click Save — ContextQA automatically adds the test description and reproduction steps to the work item.
A direct link to the Azure Boards ticket appears immediately after creation. See Azure DevOps Integration for setup instructions.
Get release readiness reports your stakeholders understand. Book a Demo → — See the analytics dashboard, failure analysis, and flaky test detection for your test suite.
Last updated
Was this helpful?