> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/testing-with-ai-coding-agents/end-to-end-testing.md). # End-to-End Testing with AI Agents Use Claude Code, Cursor, or Codex to build Playwright end-to-end tests, verify real outcomes, and run equivalent ContextQA workflows through MCP. ## Quick answer Choose one important user journey, define its starting state and final business outcome, and ask your coding agent to write a browser test with explicit assertions. Run it against an approved environment, inspect the evidence, and verify the underlying state when the outcome requires it. With ContextQA MCP, describe the same journey, review the generated steps, execute the case, and retrieve actual result details. A completed sequence of clicks is not enough. Start with a narrow flow such as creating and reopening a note. A successful toast does not prove the note was saved. Likewise, a booking thank-you page does not prove a calendar invitation or CRM meeting exists. ## Prerequisites and environment contract You need an approved staging application, an isolated test account, known seed data, a cleanup mechanism, and the correct browser runner. Retain the repository's existing Playwright version and lockfile. For a new learning project, follow [Playwright installation](https://playwright.dev/docs/intro). The example below is an **illustrative application contract**, not a runnable test against ContextQA or a supplied demo app. Adapt it to an application you own before execution: * An authenticated user can open `/notes/new`, fill fields labelled `Title` and `Body`, and click `Save note`. * That action calls `POST /api/notes`, returns 201 and JSON with `id`, then shows the notes list. * Each saved note has `data-testid="note-"` and displays its title. * Authenticated `GET /api/notes/` returns the saved `title` and `body`; `DELETE` removes only that note and returns 204. * `TEST_BASE_URL` identifies the approved test environment; `TEST_AUTH_STATE` points to an existing authenticated Playwright storage-state file for a disposable test account. Do not commit the storage-state file: it can contain reusable authentication material. Use your project's documented authentication setup and ignored storage location. [Playwright authentication](https://playwright.dev/docs/auth). ## 1. Give the agent an outcome, not only steps ``` Inspect our app, existing Playwright configuration, and test fixtures. Create an end-to-end test for an authenticated user creating a note. Use a unique title and known body, save it through the UI, verify the creation response, reload, and confirm it remains visible. Independently read the stored note through the documented API and compare its content. Clean up only this test's note, including when assertions fail. Use our approved staging target and test account. Do not send messages, charge cards, or create production records. Use semantic locators and explicit expected results. Do not use fixed sleeps, weaken assertions, or treat a toast as proof of persistence. Run the test. Report the command, environment, final assertions, result artifacts, retries, and any failed or blocked step. Preserve the first failure if a later attempt passes. ``` Use the appropriate client setup for [Claude Code](/testing-with-ai-coding-agents/claude-code-testing.md), [Cursor](/testing-with-ai-coding-agents/cursor-testing.md), or [Codex](/testing-with-ai-coding-agents/codex-testing.md). An agent can generate a test before it has an environment capable of running it; require that distinction in its handoff. ## 2. Configure the target explicitly In a learning project with `@playwright/test` installed, this minimal `playwright.config.ts` rejects an unspecified target or session: ```ts import { defineConfig } from '@playwright/test'; if (!process.env.TEST_BASE_URL || !process.env.TEST_AUTH_STATE) { throw new Error('Set TEST_BASE_URL and TEST_AUTH_STATE for the test account'); } export default defineConfig({ testDir: './tests', retries: 0, use: { baseURL: process.env.TEST_BASE_URL, storageState: process.env.TEST_AUTH_STATE, browserName: 'chromium', trace: 'retain-on-failure', }, }); ``` Merge these concepts into an existing configuration rather than replacing its projects, fixtures, or CI settings. This example assumes cookie-based authentication: `page.request` shares the browser context's cookies for API readback and cleanup. A token stored only in localStorage does not automatically authenticate those requests; configure the API's approved authentication mechanism separately. Keep session files out of version control. The example retains failure traces and starts without automatic retries so initial failures remain obvious. [Playwright configuration](https://playwright.dev/docs/test-configuration), [trace viewer](https://playwright.dev/docs/trace-viewer). ## 3. Write assertions around the business outcome Save this adapted example as `tests/notes.spec.ts`: ```ts import { randomUUID } from 'node:crypto'; import { test, expect } from '@playwright/test'; test('created note survives reload and matches stored content', async ({ page }) => { const title = `e2e-note-${randomUUID()}`; const body = 'A deterministic body for persistence verification.'; let createdId: string | undefined; try { await page.goto('/notes/new'); await page.getByLabel('Title', { exact: true }).fill(title); await page.getByLabel('Body', { exact: true }).fill(body); const responsePromise = page.waitForResponse((response) => new URL(response.url()).pathname === '/api/notes' && response.request().method() === 'POST'); await page.getByRole('button', { name: 'Save note', exact: true }).click(); const response = await responsePromise; expect(response.status()).toBe(201); const created = await response.json(); expect(typeof created.id).toBe('string'); expect(created.id.length).toBeGreaterThan(0); createdId = created.id; await expect(page).toHaveURL(/\/notes\/?$/); await page.reload(); await expect(page.getByTestId(`note-${createdId}`)).toContainText(title); const persisted = await page.request.get(`/api/notes/${encodeURIComponent(createdId!)}`); expect(persisted.status()).toBe(200); expect(await persisted.json()).toMatchObject({ id: createdId, title, body }); } finally { if (createdId) { let cleanupError: unknown; try { const cleanup = await page.request.delete(`/api/notes/${encodeURIComponent(createdId)}`); if (cleanup.status() !== 204) cleanupError = `cleanup returned ${cleanup.status()}`; } catch (error) { cleanupError = error; } expect.soft(cleanupError, 'delete only this test note').toBeUndefined(); } } }); ``` Set the two environment variables in your test environment, install the configured browser if necessary, and run: ```bash npx playwright install chromium npx playwright test tests/notes.spec.ts ``` The cleanup assertion is soft so a cleanup problem is recorded without replacing an earlier application failure. Playwright's browser-context request client can share the browser session's cookies, which makes the independent API read useful for an authenticated workflow. It still verifies only the state exposed by that API; it does not prove a downstream warehouse or third-party sync. [API testing](https://playwright.dev/docs/api-testing). ## 4. Add meaningful failure cases Start with cases suggested by the application's requirements: a required title left empty; an expired session; permission revoked before saving; a save failure that preserves the user's input; and retry after a network interruption without duplicate creation. Specify the expected outcome before asking the agent to implement each case. Keep data independent between runs. A unique title avoids collisions, while a stored record ID makes cleanup precise. If creation succeeds but the response is lost, cleanup may need a test-run identifier and a bounded server-side lookup; document that limitation instead of deleting all notes in the account. Use locators that describe the intended control. Playwright recommends resilient locators and assertions that wait for the expected UI state. Avoid converting a failing exact assertion into a broad text match just to make the run pass. [Best practices](https://playwright.dev/docs/best-practices), [assertions](https://playwright.dev/docs/test-assertions). ## 5. Run the same outcome with ContextQA MCP [Connect ContextQA MCP](/mcp-server/installation-and-setup.md), verify the intended workspace, and inspect the connected tools. The documented `create_test_case` accepts `url` and `task_description`; describe the actions **and** their expected results. Inspect generated steps before starting execution. [Case management reference](/mcp-server/tool-reference/test-case-management.md). ``` Prepare a ContextQA test for our approved staging notes workflow. Use the dedicated account and unique test-run identifier. Create a note through the UI, verify its exact title after reload, and verify persisted content using the approved read-back mechanism. Clean up only that note. Show the generated steps and assertions first. If the available tools cannot perform the API read-back or cleanup, identify that gap and use our approved companion check; do not mark those assertions passed. Execute the reviewed case and retrieve the result ID and per-step results. Report the saved-state evidence separately from UI navigation. ``` Use the documented `execute_test_case`, `get_execution_status`, and result tools. A `STATUS_COMPLETED` response means execution finished; inspect `get_test_case_results` and step evidence for the assertion verdict. Keep failed, blocked, and unverified outcomes distinct. [Execution and results reference](/mcp-server/tool-reference/execution-and-results.md). ## Debugging, retries, and portability Inspect the first failing assertion, trace, relevant request, and seeded state before changing the test. A timeout might mean the element never appeared, the wrong account was used, or the app failed; more waiting does not distinguish those causes. Preserve cleanup failures as separate operational problems when they accompany an application failure. If retries are enabled later, retain each attempt. Playwright distinguishes tests that pass initially from those that fail and subsequently pass; a retry should not erase that signal. [Retry behavior](https://playwright.dev/docs/test-retries). For ContextQA-generated browser cases, follow [export and portability](/testing-with-ai-coding-agents/export-and-portability.md). Inspect exported assertions, authentication, API checks, cleanup, and environment configuration, then rerun the project. An exported file is not evidence that the exported test passes or reproduces every managed-platform behavior. ## Common questions ### Should every feature have a long end-to-end scenario? No. Keep broad input combinations in faster tests and select important cross-system journeys for browser coverage. Short, independent journeys are easier to diagnose. ### Can ContextQA replace all Playwright fixtures automatically? Do not assume that. Evaluate the actual migrated or exported artifact, including custom setup, authentication, API state checks, and assertions. **Next step:** [Set up ContextQA MCP](/mcp-server/installation-and-setup.md) and try one reviewed staging journey. To compare native Playwright evidence with an MCP-driven workflow, [book an end-to-end testing demo](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=end-to-end-testing). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://learning.contextqa.com/testing-with-ai-coding-agents/end-to-end-testing.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.