Tutorial: Your First Test with an AI Agent

Connect an AI assistant to ContextQA MCP, review a browser test, execute it in a controlled environment, and verify its actual result and evidence.

Quick answer

Connect your AI assistant to ContextQA MCP and authenticate, then search existing cases before creating a managed browser test. Review its steps and expected outcomes, execute the agreed case in a controlled environment, and retrieve the actual verdict and available evidence. A created case, monitoring URL, or completed job does not establish that the assertions passed.

What this page covers

Who is this for? Developers and SDETs who want to drive ContextQA from an AI assistant. You connect your client to the MCP server, ask the agent to build a test case in plain English, run it, and review the evidence — all from a chat prompt.

ContextQA MCP lets a compatible AI assistant create and run managed application tests through the tools available to your account. You can describe a supported workflow in natural language without writing a test script or wiring direct API calls. Review what the tools actually created and executed; some requirements may need a companion API or repository test.

In this tutorial you will:

1. Connect the hosted MCP server to your AI assistant
2. Verify the connection with a read-only prompt
3. Create a test case from a natural-language description
4. Run the test and poll for the result
5. Review the evidence and interpret a failure

Target outcome: A reviewed case, an actual execution result, and evidence showing which assertions were checked. If access, test data, or an assertion is unavailable, record that gap instead of declaring the tutorial complete.

Prerequisites

• A ContextQA account with at least one workspace (sign up at contextqa.com)
• An MCP-compatible client installed: Claude Code, Claude Desktop, Cursor, ChatGPT, or Codex
• An approved application test environment reachable by the configured runner, with permission to perform the described actions
• A configured test-user fixture, a synthetic report with known expected values, and a way to reset test data
• Familiarity with core concepts — specifically test cases, steps, and executions

This tutorial uses Claude Code for the connection example. Adapt the prompts to the tools and permissions exposed by your client; configuration and approval behavior differ. See Installation & Setup for other clients and testing with coding agents for repository-native unit, integration, and browser workflows.

---

Step 1: Connect the MCP server to your agent

The server is hosted at https://mcp.contextqa.com/mcp. There is nothing to download or run locally — you point your client at the URL and sign in through your browser.

In your terminal, add the server to Claude Code:

claude mcp add --transport http contextqa https://mcp.contextqa.com/mcp

This registers a server named contextqa that uses HTTP transport and OAuth 2.0 authentication.

Verify configuration: Run claude mcp list and confirm contextqa appears. A listed configuration is not yet proof that an authenticated tool call succeeds.

Using a different client? Each client has its own configuration, authentication, and workspace controls. Use Installation & Setup for the supported connection path to https://mcp.contextqa.com/mcp.

---

Step 2: Sign in and verify the connection

In an interactive Claude Code session, open /mcp, select ContextQA, and use its authentication action. Complete the OAuth flow offered by the client rather than assuming the first tool call will open a browser. See Claude Code's official MCP authentication guidance.

1. Sign in with your ContextQA account.
2. Authorize the client's access when prompted.
3. Wait for the browser to auto-redirect back to your client.

Don't close the browser or navigate away until the redirect completes. If you interrupt the callback, authorization fails and you must start sign-in again. See Authentication for the full flow.

After authentication, ask your agent:

Inspect the connected ContextQA tool names and input schemas. Use a read-only
call to list recent cases in the intended workspace. Show returned IDs and
report any authentication or access error. Do not create or execute tests.

Verify access: Inspect the actual successful tool response, including its workspace scope when available. An empty successful listing is valid; an error or the agent's unsupported summary is not. Confirm the account and workspace before changing anything. For login failures, use MCP troubleshooting.

---

Step 3: Create a test case from a prompt

First search for an existing report test and inspect its steps. Reuse its ID when it already covers the behavior. Before choosing a creation or generation tool, inspect its schema and side effects: some workflows generate and execute together. For this tutorial, select a path that allows reviewing the case before execution.

Ask your agent:

Search our test workspace for an existing Reports smoke case and inspect its
steps. Use [approved test URL] and our configured test-user fixture; do not
print or paste the fixture's credentials. If a matching case exists, show
its ID and any proposed changes before updating it. Otherwise create one
case named "Reports - Smoke Test" with these reviewed requirements:
1. Sign in through the approved test fixture.
2. Open Reports and locate [synthetic report identifier].
3. Assert its exact expected title, period, and status from our seed data.
Return the case ID and complete steps. Do not execute yet.

Replace the bracketed values with approved environment and fixture references. Supply secrets through your configured test environment, not conversational text. Checking a known report is stronger than accepting any report in a possibly stale list. Inspect the resulting locators, steps, and expected values before accepting the case.

Verify the case: Retrieve its steps using the returned test_case_id and compare them with the approved requirements. Confirm the target URL, fixture, report identity and assertions. Keep the ID for the next step; a creation response alone is not execution evidence.

Tip: If the steps are incomplete, propose a correction to the existing case ID, review the change, and update that case through the supported tools. Do not repeatedly recreate it and leave duplicate cases behind.

---

Step 4: Run the test and wait for the result

Execution is asynchronous. Confirm the configured environment and test fixture, then start one run of the reviewed case. The documented flow uses execute_test_case, get_execution_status, and the result tools; exact parameters and returned fields come from your connected schema. A hosted runner cannot automatically reach an application that is accessible only on your laptop.

Ask your agent:

Execute reviewed test case [case ID] once in the approved environment.
Retain the returned execution identifiers. Monitor with a bounded wait and
the server's recommended polling interval. Retrieve the final assertion
verdict and step results for this run. Do not substitute an older result.
Report blockers, skipped checks and unavailable evidence explicitly.

The agent:

1. Starts the reviewed case and retains the tool response and execution identifiers.
2. Monitors status using the connected schema without launching duplicate runs.
3. Retrieves get_test_case_results and relevant step details for the returned result or execution ID.
4. Reports actual assertion outcomes and any execution or environment problem separately.

If a run exceeds the agreed waiting period, report it as still running or unresolved. Inspect its status before retrying; another run could duplicate activity and make a “latest result” lookup ambiguous.

Verify the verdict: Check the actual result and relevant assertions, including skipped or unsupported checks. A STATUS_COMPLETED job status means execution finished; it is not itself an assertion verdict. See execution and results.

Status	What it means
Passed verdict	The executed assertions report success; confirm they cover the intended requirements
Failed verdict	Inspect the failed assertion and evidence before diagnosing the cause
Completed job	Execution ended; retrieve its actual verdict and step results
Blocked or unavailable evidence	Report the missing check; do not infer a pass
Still running at the wait deadline	Preserve the run ID and report the unresolved state

---

Step 5: Review the results and diagnose a failure

Available evidence depends on the runner, enabled features, account permissions and the execution itself. Request step results and any available screenshots, recordings, logs or traces. Do not promise an artifact bundle that the tool did not return. Retain the run identifier with the evidence and use the applicable access and retention policy when sharing links.

To review a passing run, ask:

Retrieve the step results for [result or execution ID]. Show which assertions
ran and their observed outcomes. Link available evidence for that same run;
list missing artifacts or checks instead of inventing them.

Use the available result and step-detail tools, interpreting the fields actually returned. Verify the known report identity and expected values rather than accepting successful navigation as proof.

To diagnose a failure, ask:

For [failed result ID], identify the first failed assertion and compare its
expected and actual values. Retrieve available diagnostic evidence. Separate
observed facts from possible causes, and propose the next discriminating check.
Do not modify the test or apply a self-healing suggestion yet.

If a diagnostic tool such as get_root_cause is available, treat its explanation as a hypothesis until the evidence supports it. An unavailable report could reflect a missing fixture, the wrong account, an application defect or an incorrect locator; the same visible failure can have several causes.

Review repairs: Inspect any proposed locator or self-healing change before applying it. Preserve the intended report and assertions; changing the expected value or selecting a different report to obtain a green run changes coverage. Keep the original failure, apply only the reviewed repair to the existing case, then record the new execution separately. See the Agent Integration Guide for tool orchestration.

---

Summary

Before calling this workflow complete, confirm that you:

1. Connected the hosted MCP server to your client with a single command
2. Signed in through OAuth 2.0 and verified the connection with a read-only prompt
3. Created or reused a test case and reviewed its steps and assertions
4. Ran the test and let the agent poll for the result
5. Reviewed the actual verdict and available evidence, preserving unresolved gaps

Tool availability remains specific to the connected account and server. Keep native unit and integration results separate from managed application execution. For a framework-owned follow-up, use export and portability and verify the resulting code before adding it to CI.

Next steps

• Generate tests from your existing artifacts: Ask the agent to create tests from a Jira ticket, an OpenAPI spec, a Figma design, or a git diff. See AI Test Generation.
• Review a bug-fix loop: Reproduce a failure, inspect evidence, review the proposed repair, and rerun with the original assertions preserved. See Agent Integration Guide.
• Plan CI execution: Review authentication, environment, cleanup and workflow triggers before automating runs. See GitHub Actions.
• Add repository tests: Use the coding-agent testing guides for unit, integration, functional and end-to-end testing.
• Explore available tools: Browse the Tool Reference, then verify exact names and parameters in the connected manifest.

Related pages

• MCP Server Overview — what the MCP server is and what it exposes
• Installation & Setup — connect Claude Desktop, Cursor, ChatGPT, and Codex
• Authentication — the OAuth 2.0 sign-in flow, sessions, and multiple accounts
• Agent Integration Guide — tool ordering, async execution, and multi-step orchestration
• Tool Reference — full parameter documentation for every tool

Evaluate one real application workflow with your AI assistant. Book a testing workflow demo with your acceptance criteria and approved test environment.