Test AI-Generated Code

Validate AI-generated code against independent acceptance rules, detect deliberate defects, and check changed application journeys with ContextQA MCP.

Quick answer

Test AI-generated code against a behavior contract written independently of the implementation. Check the changed rule, denied paths, and boundary values; then prove a deliberate defect makes the relevant assertion fail. Run repository tests locally or in CI, and verify the deployed application separately. ContextQA MCP can support that application check from your coding agent, while your existing runner remains responsible for native unit tests.

This guide is for developers reviewing changes from Claude Code, Cursor, Codex, or another coding assistant. It concerns ordinary application code authored by AI. Evaluating an LLM application's generated answers requires additional checks for output quality, variability, and misuse; a passing JavaScript suite does not establish those properties.

Why generated tests can agree with a bug

An implementation and its generated tests can share the same mistaken assumption. For example, an export endpoint might check that a user has download permission but forget that the report must belong to the same tenant. A test that merely repeats the permission expression can pass while preserving that defect.

Separate the source of expected behavior from the code under review. Use an approved acceptance criterion, API contract, incident reproduction, or stakeholder decision. If the rule is ambiguous, record the unresolved question instead of letting the assistant silently choose a policy. Asking a second model to review the same code helps generate ideas, but does not by itself supply an independent expected result.

1. Turn the requirement into explicit decisions

Consider an illustrative report-download feature. Its contract is:

• The authenticated principal has a non-empty tenant identifier and explicit export permission.
• The report belongs to that tenant and has status ready.
• Access expires at the specified time: a request at the exact expiration is denied.
• Missing or malformed inputs are denied.

Write these decisions before inspecting a proposed implementation:

Scenario	Expected decision	Defect this can expose
Authorized principal, same tenant, ready, before expiry	Allow	Accidental denial of valid downloads
Authorized principal, different tenant	Deny	Missing tenant boundary
Same tenant, exactly at expiry	Deny	Incorrect inclusive boundary
Same tenant, still building	Deny	Download before report completion
Same tenant, no export permission	Deny	Missing permission check

Keep a successful case as well as denied cases. A function that always returns false is safe from these particular unauthorized downloads but still violates the feature contract.

2. Review the agent's plan before the patch

Give the agent the requirement, changed files, existing runner command, and a focused task:

Review this report-download change against the supplied contract.
First list acceptance cases independently of the implementation.
Identify any unresolved rules; do not invent the expected behavior.

Add tests using our existing runner. Include the allowed case,
different-tenant denial, missing permission, and exact expiry boundary.
Import application code rather than reproducing its logic in tests.
Do not change expected results, snapshots, or permission rules just to
make the suite pass. Preserve existing failures and skipped-test counts.

Show which assertion rejects a temporary removal of the tenant check.
Restore that edit. Report the original and final commands and results.

Review the diff for deleted assertions, broad mocks, swallowed errors, and changed fixtures. A test that replaces the permission function with a mock returning true cannot establish that the real permission function works. Keep a record of failures found before any retry or repair.

3. Run a small, independently specified example

Create an empty local folder using Node.js 24 or later. No packages, account, or repository download are required. Save this implementation as download-policy.mjs:

// Teaching fixture: authentication and report lookup happen outside this function.
export function canDownload(principal, report, nowMs) {
  if (!principal || !report || !Number.isFinite(nowMs)) return false;
  if (typeof principal.tenantId !== 'string' || principal.tenantId.length === 0) return false;
  if (!Array.isArray(principal.permissions) || !principal.permissions.includes('reports:export')) return false;
  if (principal.tenantId !== report.tenantId || report.status !== 'ready') return false;
  return Number.isFinite(report.expiresAt) && nowMs < report.expiresAt;
}

Save the following as download-policy.test.mjs in the same folder. Its literal expected decisions come from the contract above:

import test from 'node:test';
import assert from 'node:assert/strict';
import { canDownload } from './download-policy.mjs';

const principal = { tenantId: 'tenant-a', permissions: ['reports:export'] };
const report = { tenantId: 'tenant-a', status: 'ready', expiresAt: 2000 };

// Expected decisions come from the documented contract, not from another
// implementation of the same permission expression.
const cases = [
  ['allows an authorized ready report before expiry', principal, report, 1999, true],
  ['denies a different tenant despite export permission', principal, { ...report, tenantId: 'tenant-b' }, 1999, false],
  ['denies at the exact expiry time', principal, report, 2000, false],
  ['denies after expiry', principal, report, 2001, false],
  ['denies a report that is still building', principal, { ...report, status: 'building' }, 1999, false],
  ['denies an actor without export permission', { ...principal, permissions: [] }, report, 1999, false],
  ['denies a missing principal', null, report, 1999, false],
  ['denies missing tenant identity on both inputs', { ...principal, tenantId: '' }, { ...report, tenantId: '' }, 1999, false],
  ['denies a missing report', principal, null, 1999, false],
  ['denies a non-numeric expiration', principal, { ...report, expiresAt: '2000' }, 1999, false],
  ['denies a non-finite clock value', principal, report, NaN, false],
];

for (const [name, actor, target, nowMs, expected] of cases) {
  test(name, () => assert.equal(canDownload(actor, target, nowMs), expected));
}

Run from that folder:

node --test download-policy.test.mjs

The suite evaluates 11 contract decisions. To check its sensitivity, make a backup of download-policy.mjs, temporarily remove principal.tenantId !== report.tenantId || from its fourth condition, and rerun. The cross-tenant assertion should fail. Restore the backup; separately change nowMs < report.expiresAt to nowMs <= report.expiresAt and confirm the exact-expiry assertion fails. Restore the original file again and confirm all tests pass. Keep these defects in a disposable local example, never a shared deployment.

Check the actual assertion failure rather than accepting any nonzero exit code: a syntax error does not prove fault detection. Node's built-in test runner provides the execution mechanism. Node test runner documentation.

This fixture checks a pure decision function. It does not test authentication, database lookup, HTTP authorization, or an actual download. In production, derive the principal from trusted authentication and report ownership from trusted storage, rather than accepting those fields from a request body.

4. Use coverage and mutation testing for different questions

Coverage helps locate code the suite did not exercise. Executing a branch does not prove its result was meaningfully checked. Mutation testing changes production logic temporarily and asks whether tests reject that change. Stryker automates this process for supported language and runner combinations. Stryker mutation testing introduction.

Start with the changed permission or calculation module. Review survivors individually: a missing boundary assertion, an unreachable case, and a behaviorally equivalent change require different responses. An equivalent mutant cannot be distinguished by an input that preserves the program's observable behavior; chasing an arbitrary 100% score can waste effort. Stryker equivalent mutants.

Record the file scope, runner, exclusions, and outcomes. Keep killed mutants, survivors, timeouts, and invalid runs separate when reviewing evidence; a process error is not an assertion catching the intended defect. The two deliberate checks above are useful demonstrations, not a Stryker run or a measured mutation score. Stryker states and metrics.

5. Check whether the application actually enforces the rule

Add an integration test that exercises the real endpoint with controlled authenticated identities. Prove that a direct request for another tenant's report returns the documented denial response and no report bytes. A hidden Download button alone does not establish server-side authorization.

Then check the visible workflow: an authorized user downloads the correct report, while an unauthorized user receives the expected explanation. Verify downloaded content or a controlled backend record where supported; a success toast is insufficient evidence. See the end-to-end testing guide for separating browser assertions from backend outcomes.

6. Add the ContextQA MCP application handoff

Follow MCP installation and setup, authenticate, and inspect the tools available to your workspace. Use a disposable staging report and approved test identities. The hosted MCP server does not automatically see a local checkout or execute its Node suite.

After discovery, adapt this prompt to supported tools:

Find an existing report-download test before creating another.
Use the supplied staging URL and approved test identities.
Prepare the authorized download and denied cross-tenant scenarios.
Show the test steps and exact expected outcomes for review.

Execute the reviewed cases, then retrieve their final results.
Report case ID, run ID, build/environment, each assertion outcome,
and available evidence. Mark unsupported direct-request or file-content
checks as unverified; do not substitute a hidden button or success toast.

Keep repository-test results and application-run results separate. If a requested check is unavailable through the connected tools, retain it in your API or Playwright suite. Before moving a managed test into CI, follow the export and portability checklist and verify the exported assertions independently.

Acceptance record for the pull request

Attach the contract, changed behavior, exact test commands, detected defects, application build/run identifiers, and unresolved checks. Distinguish passed, failed, skipped, and not run. This gives a reviewer evidence to assess instead of a blanket claim that the agent tested everything.

Frequently asked questions

Is high coverage enough to approve AI-generated code?

No. Review what assertions establish, especially denied paths and boundaries. Coverage and a focused defect-detection check answer complementary questions.

Should a different agent write the tests?

That can provide another perspective. Independence comes primarily from the expected behavior and evidence source, not the assistant's name. Review the contract before either agent changes code.

Does ContextQA MCP replace my unit-test runner?

No. Keep native tests in their established runner. Use MCP for supported application testing tasks and retrieve actual run results before claiming those checks passed.

Try this on one change

Choose one risky rule in your next AI-assisted pull request. Establish its expected decisions, demonstrate one detected defect, and connect the affected application flow using the ContextQA MCP setup guide. For help designing that evaluation, book a ContextQA demo.

Sources reviewed: September 25, 2026. The examples are teaching fixtures; they do not represent a live ContextQA execution or an audit of your application.