Failure Analysis
Review failed-test evidence, distinguish observed defects from blocked runs, assess root-cause hypotheses, and prepare a reproducible engineering handoff.
Quick answer
Start with the original execution and its first relevant failed assertion. Check the application build, role, data and available artifacts before classifying the result as a product, test, environment or dependency problem. Treat AI root-cause suggestions as hypotheses to review; an unrelated failed step or a passing retry does not establish whether the reported defect exists.
For a ticketed bug, use the support-to-engineering reproduction workflow to record execution status separately from the reproduction conclusion.
Triage an overnight run without rerunning first
Review the original evidence before spending a retry. Create one record per failing case:
Run identity
Plan, build, environment, browser, start time, and result ID
First failed step
Observed error and exact assertion
Evidence
Screenshot, video timestamp, network/console evidence, and trace link
AI hypothesis
Root-cause suggestion, clearly labeled as a hypothesis
Human check
Evidence that supports or contradicts the hypothesis
Classification
Product, test, data, environment, dependency, or suspected flake
Disposition
Fix, quarantine with replacement coverage, investigate, or accept risk
Ownership
Named owner, due date, and linked ticket
Cluster failures only after confirming they share a cause. Twenty tests failing after one authentication outage are one incident with twenty affected cases, not twenty independently proven product defects.
Rerun only when the record states what new information the rerun will provide. Preserve the original failure and report both attempts so a passing retry cannot erase the release signal.
When a test fails, the goal is to determine as fast as possible whether the failure represents a real bug in the application, a flaky environmental issue, or a test that needs updating. ContextQA provides several tools to answer that question without manual log analysis.
Interactive course: analyze root cause and impact
Use these two course segments to move from a failed execution and its captured evidence to an AI-assisted root-cause and impact assessment.
Part 1 — Inspect the failed execution
Part 2 — Review AI root cause and impact
What to notice
The investigation starts with the first meaningful failure and the evidence actually captured by that run.
AI analysis distinguishes likely product, script, data, environment, and integration causes without presenting an unsupported conclusion as fact.
Impact context helps a reviewer decide whether to update a test, rerun it, investigate the application, or create a defect.
Accessing failure analysis from the results page
After a test execution completes:
Open the execution result page (via View Detailed Report in the post-run pop-up, or via Execution Dashboard → run history).
Click the failed step or the failing test case card.
The failure detail panel expands showing the error message, screenshot, and network log entries captured at the moment of failure.
From this panel you can access the full Root Cause Analysis view by clicking the root cause indicator or navigating to the failure analysis section.
Root cause analysis
ContextQA presents the current analysis in the Failure Insights bar on the run result. It summarizes the likely cause, category, diagnosis confidence, and available next action. Use View reasoning to inspect why the evidence supports the diagnosis; when a repair is available, use Review fix or Review improvement before changing the test.
The structured analysis can contain:
Failure category — a classification of the failure type (e.g., element not found, assertion mismatch, network error, timeout)
AI explanation — a plain-English description of what went wrong and why, generated from the screenshot, DOM state, and error message captured at the point of failure
Fix suggestion — a recommended action to resolve the failure, such as updating a locator, adding a wait step, or investigating a specific API endpoint
The AI explanation and fix suggestion are displayed directly in the result page so a developer or QA engineer can act on them without needing to reproduce the failure locally first.
Evidence used by root cause analysis
The analysis can correlate several evidence layers, but only evidence captured by that run is available:
Failed step and runtime error
For a recorded failed step.
Screenshot and DOM/page state
For supported browser runs when visual and page-state capture completed.
Console and network data
When the browser run captured those logs; API-only and interrupted runs can have different evidence.
Test data and configuration context
When the run used attached data or environment configuration. Secret values remain masked and should not be copied into a defect.
Historical pattern
When earlier comparable executions exist. A single run cannot establish a repeatable trend by itself.
Repository file or line
Only when a compatible code/repository integration supplies source context. It is not guaranteed for every failure.
The output is best-effort decision support. Depending on the available evidence, it can include a category, explanation, affected step, related failure pattern, suggested fix, and—when source correlation is available—a candidate file or line. Review the evidence before changing a test or filing a product defect.
When you create a Jira or Azure DevOps defect, ContextQA pre-populates the fields supported by the connected tracker from the evidence it has, such as the test name, failing step, error, root-cause summary, screenshot or evidence link, and result URL. Review the draft and remove sensitive data before submitting it; missing evidence is not fabricated.
Root cause categories
The AI classifies each failure into one of the following categories. The category tells you where the problem lies and who should act on it:
Script Failure
The automation script failed to execute as written — a locator, wait, or step in the test broke. Fix the step and re-run.
Functional / Product Bug
The application under test behaved incorrectly. A genuine product defect — report it to the development team.
Test Data Issue
Invalid, missing, or stale test data caused the failure. Update the test data and re-run.
Environment / Infrastructure
A network, server, or deployment problem caused the failure. Check the environment and retry.
ContextQA Platform
The failure originated in the ContextQA platform itself. Contact ContextQA support if it persists.
Flaky / Intermittent
A potentially unstable, non-deterministic failure. Confirm the pattern across comparable runs, then fix its cause rather than hiding it with unconditional retries.
Configuration / Setup
A misconfiguration or incomplete setup caused the failure. Review the test or application configuration and prerequisites.
Dependency / Integration
A failure in an external dependency or integration. Verify the dependent API or third-party service.
Overlay / Pop-up Blocker
A cookie banner, modal, or other overlay prevented the intended interaction. Review the captured page state and decide whether the blocker belongs in the test setup, application behavior, or environment before labeling it a product defect.
Unclassified / Needs Review
The failure couldn't be automatically categorized and requires manual review.
An overlay classification prevents a blocker from being presented as a generic application bug when the evidence shows that another visible layer intercepted the action. It remains a hypothesis until a reviewer confirms the screenshot, DOM state, and intended user behavior.
Debug mode for interactive diagnosis
ContextQA includes a Debug mode that lets you diagnose failures interactively as a test runs. When a step fails during a debug session, you are presented with options to handle it without stopping the run:
Enter more detail — provide a plain-English instruction to clarify the step (e.g., "click on admin")
Update and Rerun — modify the failing step and retry from that point
Add a New Step and Rerun — insert a new step at the failure point and resume
Skip Step — bypass the failed step and continue with the remaining steps
Delete Step — remove the step from the test case entirely
This workflow is useful during test authoring and maintenance when you want to iterate on a test case in real time rather than running it repeatedly from scratch.
Suggested fixes
When the auto-fix capability is enabled, ContextQA does more than explain a failure — it proposes concrete changes to the test and lets you apply them without opening the step editor. This turns root cause analysis into a review-and-apply repair workflow for script-level failures.
Unlike AI self-healing, which repairs locator drift automatically while a test runs, suggested fixes are proposed after a run fails and applied by you after review. A suggested fix can update a step, add a step, or remove a step — not only change a locator — and always requires a re-run to confirm the fix works.
Opening the suggested fixes panel
On the execution result page for a failed run:
Open the failed run's Analysis area and find Failure Insights.
Select View reasoning to inspect the evidence-backed diagnosis.
Select Review fix, Review improvement, or View instructions when that action is available.
After a change, use Rerun, Watch live, and View run result to follow the verification execution. The result keeps the original failure and the rerun as separate evidence.
Types of proposed change
Each proposed fix targets one step and falls into one of three change types:
Update a step
Proposed change
Modifies an existing step — for example, replaces a broken locator with a Proposed locator plus one or more Fallback locators, or updates a data value.
Add a step
New step
Inserts a new step at the failure point.
Remove a step
Step removal
Deletes a step that is no longer valid.
Some proposed changes require a concrete value from you, such as the destination URL for a newly added navigation step. Enter the requested value in the fix panel before applying it. ContextQA writes that value into the action field the step uses instead of saving a placeholder; review the resulting step and re-run it as you would any other applied fix.
Applying a fix
Apply fix — applies the proposed change for a single step. Use this to review and apply changes one at a time.
Apply all fixes — applies every not-yet-applied step in one operation. Steps that are already applied are skipped.
After a fix is applied, ContextQA confirms with Step updated. Re-run the test to verify. (or Steps updated. Re-run the test to verify. for multiple steps). The applied steps are marked as fixed and their Apply fix control is hidden. When every proposed step has been applied, the run is marked as fixed and the top-of-report button switches to Fix applied · See fix.
Other actions in the panel
Beyond applying fixes, the panel lets you:
Jump to failed step — scrolls to and highlights the failing step in the step list.
Re-run — re-executes the test case to verify a fix.
Edit manually — opens the test case in the Test Steps Editor so you can make the change yourself instead of applying the proposal.
Mark as bug — creates a Jira bug from the failure when the root cause is a genuine product defect rather than a test issue. This requires a connected Jira integration; if none is configured, ContextQA prompts you to connect Jira from Integrations & Plugins. See Jira Integration.
AI self-healing and auto-healed steps
When the application UI changes in a way that breaks a locator, ContextQA's AI self-healing engine attempts to automatically identify the correct new element and update the step. If the accuracy score is above 90% and there is no functional change to the flow, the step is healed automatically.
Healed steps are marked with an auto-healed indicator in the execution report. This means the test passed using an updated locator — not the original one — so you should review the healed step and confirm the update is correct before relying on it long-term.
To approve or review auto-healing suggestions, use the approve_auto_healing MCP tool or navigate to the auto-healing review queue in the platform.
Flaky test detection
A test is considered flaky when it produces inconsistent results across runs on the same code and the same environment. ContextQA tracks pass/fail history per test case and surfaces tests that show non-deterministic behavior.
Flaky tests appear in the Consistently Failing list on the Execution Dashboard, often alongside a root cause note indicating intermittent failures. Use the failure trend data — available on the test case detail view — to see how often a given test has failed over recent runs and whether the failures cluster around specific times, environments, or deployments.
Common flaky failure categories surfaced by ContextQA:
Timing issues — a step executing before a page element has fully loaded
Network variability — a step depending on an API response that is occasionally slow
Data state issues — test data that is not reset between runs
Failure trends
The Consistently Failing panel in the Execution Dashboard lists test cases ranked by failure frequency. Each card shows the test case name, the failure count, and the AI-identified root cause when available.
Use this list to prioritize maintenance work. Test cases that appear repeatedly with the same root cause are strong candidates for a locator update or a stability fix. Test cases with varied root causes may indicate genuine application instability rather than test fragility.
Integrating failures with bug trackers
Once you identify a genuine application bug from a failing test, you can push it directly to your bug tracker without leaving ContextQA.
Reporting to Jira
From the test result detail page:
Click the JIRA option in the failure reporting toolbar.
Select your Jira project and set the issue type to Bug.
Click Save — a Jira ticket is created and linked to the ContextQA result.
The Jira issue is pre-populated with the test description, reproduction steps, and the result page link. See Jira Integration for setup instructions.
Reporting to Linear
From Report Bug, choose the connected Linear destination and review the prefilled title and description before submitting. Select the intended Linear project; if the connection returns no eligible project, ContextQA shows No projects available instead of creating an unscoped issue. After creation, use the ticket link in the confirmation to open the Linear issue and verify its project, content, and evidence reference.
You can also create defect tickets programmatically using the create_defect_ticket MCP tool:
Reporting to Azure DevOps
From the test result detail page:
Click Report Bug in the failure reporting toolbar.
Select Azure Board as the destination.
Choose the issue type (e.g., Bug, Test Case) and enter a title.
Click Save — ContextQA automatically adds the test description and reproduction steps to the work item.
A direct link to the Azure Boards ticket appears immediately after creation. See Azure DevOps Integration for setup instructions.
Last updated
Was this helpful?