Failure Analysis
How to use ContextQA's failure analysis, root cause, flaky test detection, and bug tracker integration features to diagnose and resolve test failures quickly.
Quick answer
How to use ContextQA's failure analysis, root cause, flaky test detection, and bug tracker integration features to diagnose and resolve test failures quickly. Use this page to understand when the capability applies, complete its user-facing workflow, and verify the expected result.
What this page covers
When a test fails, the goal is to determine as fast as possible whether the failure represents a real bug in the application, a flaky environmental issue, or a test that needs updating. ContextQA provides several tools to answer that question without manual log analysis.
Interactive course: analyze root cause and impact
Use these two course segments to move from a failed execution and its captured evidence to an AI-assisted root-cause and impact assessment.
Part 1 — Inspect the failed execution
Part 2 — Review AI root cause and impact
What to notice
The investigation starts with the first meaningful failure and the evidence actually captured by that run.
AI analysis distinguishes likely product, script, data, environment, and integration causes without presenting an unsupported conclusion as fact.
Impact context helps a reviewer decide whether to update a test, rerun it, investigate the application, or create a defect.
Accessing failure analysis from the results page
After a test execution completes:
Open the execution result page (via View Detailed Report in the post-run pop-up, or via Execution Dashboard → run history).
Click the failed step or the failing test case card.
The failure detail panel expands showing the error message, screenshot, and network log entries captured at the moment of failure.
From this panel you can access the full Root Cause Analysis view by clicking the root cause indicator or navigating to the failure analysis section.
Root cause analysis
ContextQA's AI engine analyzes failed steps and produces a structured root cause report. The report contains:
Failure category — a classification of the failure type (e.g., element not found, assertion mismatch, network error, timeout)
AI explanation — a plain-English description of what went wrong and why, generated from the screenshot, DOM state, and error message captured at the point of failure
Fix suggestion — a recommended action to resolve the failure, such as updating a locator, adding a wait step, or investigating a specific API endpoint
The AI explanation and fix suggestion are displayed directly in the result page so a developer or QA engineer can act on them without needing to reproduce the failure locally first.
Evidence used by root cause analysis
The analysis can correlate several evidence layers, but only evidence captured by that run is available:
Failed step and runtime error
For a recorded failed step.
Screenshot and DOM/page state
For supported browser runs when visual and page-state capture completed.
Console and network data
When the browser run captured those logs; API-only and interrupted runs can have different evidence.
Test data and configuration context
When the run used attached data or environment configuration. Secret values remain masked and should not be copied into a defect.
Historical pattern
When earlier comparable executions exist. A single run cannot establish a repeatable trend by itself.
Repository file or line
Only when a compatible code/repository integration supplies source context. It is not guaranteed for every failure.
The output is best-effort decision support. Depending on the available evidence, it can include a category, explanation, affected step, related failure pattern, suggested fix, and—when source correlation is available—a candidate file or line. Review the evidence before changing a test or filing a product defect.
When you create a Jira or Azure DevOps defect, ContextQA pre-populates the fields supported by the connected tracker from the evidence it has, such as the test name, failing step, error, root-cause summary, screenshot or evidence link, and result URL. Review the draft and remove sensitive data before submitting it; missing evidence is not fabricated.
Root cause categories
The AI classifies each failure into one of the following categories. The category tells you where the problem lies and who should act on it:
Script Failure
The automation script failed to execute as written — a locator, wait, or step in the test broke. Fix the step and re-run.
Functional / Product Bug
The application under test behaved incorrectly. A genuine product defect — report it to the development team.
Test Data Issue
Invalid, missing, or stale test data caused the failure. Update the test data and re-run.
Environment / Infrastructure
A network, server, or deployment problem caused the failure. Check the environment and retry.
ContextQA Platform
The failure originated in the ContextQA platform itself. Contact ContextQA support if it persists.
Flaky / Intermittent
A potentially unstable, non-deterministic failure. Confirm the pattern across comparable runs, then fix its cause rather than hiding it with unconditional retries.
Configuration / Setup
A misconfiguration or incomplete setup caused the failure. Review the test or application configuration and prerequisites.
Dependency / Integration
A failure in an external dependency or integration. Verify the dependent API or third-party service.
Unclassified / Needs Review
The failure couldn't be automatically categorized and requires manual review.
Debug mode for interactive diagnosis
ContextQA includes a Debug mode that lets you diagnose failures interactively as a test runs. When a step fails during a debug session, you are presented with options to handle it without stopping the run:
Enter more detail — provide a plain-English instruction to clarify the step (e.g., "click on admin")
Update and Rerun — modify the failing step and retry from that point
Add a New Step and Rerun — insert a new step at the failure point and resume
Skip Step — bypass the failed step and continue with the remaining steps
Delete Step — remove the step from the test case entirely
This workflow is useful during test authoring and maintenance when you want to iterate on a test case in real time rather than running it repeatedly from scratch.
Suggested fixes
When the auto-fix capability is enabled, ContextQA does more than explain a failure — it proposes concrete changes to the test and lets you apply them without opening the step editor. This turns root cause analysis into a review-and-apply repair workflow for script-level failures.
Unlike AI self-healing, which repairs locator drift automatically while a test runs, suggested fixes are proposed after a run fails and applied by you after review. A suggested fix can update a step, add a step, or remove a step — not only change a locator — and always requires a re-run to confirm the fix works.
Opening the suggested fixes panel
On the execution result page for a failed run:
Select the Failure analysis & suggested fixes button at the top of the report. After a fix has been applied, this button is replaced by a green Fix applied · See fix button (or Auto-fixed · See fix when the fix was applied automatically) that reopens the same panel.
The AI Debug tab opens and scrolls to the Root Cause Analysis panel, which shows the failure category, the AI explanation, and the proposed fix for each affected step.
Types of proposed change
Each proposed fix targets one step and falls into one of three change types:
Update a step
Proposed change
Modifies an existing step — for example, replaces a broken locator with a Proposed locator plus one or more Fallback locators, or updates a data value.
Add a step
New step
Inserts a new step at the failure point.
Remove a step
Step removal
Deletes a step that is no longer valid.
Some proposed changes require a concrete value from you, such as the destination URL for a newly added navigation step. Enter the requested value in the fix panel before applying it. ContextQA writes that value into the action field the step uses instead of saving a placeholder; review the resulting step and re-run it as you would any other applied fix.
Applying a fix
Apply fix — applies the proposed change for a single step. Use this to review and apply changes one at a time.
Apply all fixes — applies every not-yet-applied step in one operation. Steps that are already applied are skipped.
After a fix is applied, ContextQA confirms with Step updated. Re-run the test to verify. (or Steps updated. Re-run the test to verify. for multiple steps). The applied steps are marked as fixed and their Apply fix control is hidden. When every proposed step has been applied, the run is marked as fixed and the top-of-report button switches to Fix applied · See fix.
Applying a fix changes your test case immediately. Re-run the test afterward to confirm the change resolves the failure — applying a fix doesn't re-execute the test for you.
Other actions in the panel
Beyond applying fixes, the panel lets you:
Jump to failed step — scrolls to and highlights the failing step in the step list.
Re-run — re-executes the test case to verify a fix.
Edit manually — opens the test case in the Test Steps Editor so you can make the change yourself instead of applying the proposal.
Mark as bug — creates a Jira bug from the failure when the root cause is a genuine product defect rather than a test issue. This requires a connected Jira integration; if none is configured, ContextQA prompts you to connect Jira from Integrations & Plugins. See Jira Integration.
AI self-healing and auto-healed steps
When the application UI changes in a way that breaks a locator, ContextQA's AI self-healing engine attempts to automatically identify the correct new element and update the step. If the accuracy score is above 90% and there is no functional change to the flow, the step is healed automatically.
Healed steps are marked with an auto-healed indicator in the execution report. This means the test passed using an updated locator — not the original one — so you should review the healed step and confirm the update is correct before relying on it long-term.
To approve or review auto-healing suggestions, use the approve_auto_healing MCP tool or navigate to the auto-healing review queue in the platform.
Flaky test detection
A test is considered flaky when it produces inconsistent results across runs on the same code and the same environment. ContextQA tracks pass/fail history per test case and surfaces tests that show non-deterministic behavior.
Flaky tests appear in the Consistently Failing list on the Execution Dashboard, often alongside a root cause note indicating intermittent failures. Use the failure trend data — available on the test case detail view — to see how often a given test has failed over recent runs and whether the failures cluster around specific times, environments, or deployments.
Common flaky failure categories surfaced by ContextQA:
Timing issues — a step executing before a page element has fully loaded
Network variability — a step depending on an API response that is occasionally slow
Data state issues — test data that is not reset between runs
Failure trends
The Consistently Failing panel in the Execution Dashboard lists test cases ranked by failure frequency. Each card shows the test case name, the failure count, and the AI-identified root cause when available.
Use this list to prioritize maintenance work. Test cases that appear repeatedly with the same root cause are strong candidates for a locator update or a stability fix. Test cases with varied root causes may indicate genuine application instability rather than test fragility.
Integrating failures with bug trackers
Once you identify a genuine application bug from a failing test, you can push it directly to your bug tracker without leaving ContextQA.
Reporting to Jira
From the test result detail page:
Click the JIRA option in the failure reporting toolbar.
Select your Jira project and set the issue type to Bug.
Click Save — a Jira ticket is created and linked to the ContextQA result.
The Jira issue is pre-populated with the test description, reproduction steps, and the result page link. See Jira Integration for setup instructions.
You can also create defect tickets programmatically using the create_defect_ticket MCP tool:
Reporting to Azure DevOps
From the test result detail page:
Click Report Bug in the failure reporting toolbar.
Select Azure Board as the destination.
Choose the issue type (e.g., Bug, Test Case) and enter a title.
Click Save — ContextQA automatically adds the test description and reproduction steps to the work item.
A direct link to the Azure Boards ticket appears immediately after creation. See Azure DevOps Integration for setup instructions.
Last updated
Was this helpful?