> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/design-test-cases.md).

# Design AI Agent Evaluation Test Cases

Build static and dynamic AI agent tests from business requirements, personas, corrected inputs, negative scenarios, and controlled fixtures.

> **Documentation owner:** [Deep Barot](https://contextqa.com/academy/) · **Last updated:** November 11, 2026

## Quick answer

A useful AI agent test specifies a user goal, starting state, conversation conditions, expected actions, and measurable outcomes. In ContextQA, organize cases under reviewed use cases and personas. Use static cases for prescribed user turns and dynamic cases for adaptive conversations; both require explicit success criteria and controlled test data.

## Key takeaways

* Cover ordinary success, corrections, refusal, recovery, and downstream outcomes.
* Use persona variation to test interaction conditions without changing the business requirement.
* Control fixtures so parallel or repeated tests do not create accidental collisions.

## Write the business requirement first

For a scheduling agent, “book an appointment” is too broad. A testable requirement is: collect the caller’s final corrected name, service, date, time, and timezone; read back those values; obtain affirmative confirmation; then create one matching booking. If the slot is unavailable, offer alternatives without claiming the original booking succeeded.

Use this template for each case:

```
Use case: Create one appointment
Starting state: Isolated test customer; requested slot available
User goal: Book the specified service at the requested local time
Variation: User corrects the date after the first readback
Required behavior: Use the final correction and ask for confirmation
Forbidden behavior: Create before confirmation or create duplicates
Expected evidence: Conversation + matching backend record
Cleanup: Remove only the booking created by this test
```

## Choose static or dynamic conversation design

| Design  | Best use                                               | Review requirement                                                 |
| ------- | ------------------------------------------------------ | ------------------------------------------------------------------ |
| Static  | Reproduce an exact correction sequence or phrasing     | Every user turn must still make sense after the preceding response |
| Dynamic | Exercise how an adaptive simulated user pursues a goal | Bound the goal, available facts, and permitted behavior            |

Dynamic generation expands interaction variety; it does not remove the need for deterministic ground truth. Static cases improve reproducibility, but a scripted next turn can become inappropriate if the agent takes an unexpected branch. Review that mismatch before attributing the whole failure to the target.

## Build a coverage matrix

| Dimension                 | Example                                     | Required evidence                              |
| ------------------------- | ------------------------------------------- | ---------------------------------------------- |
| Normal completion         | Patient user gives one detail at a time     | Correct confirmation and completed action      |
| Combined details          | User gives name and service in one sentence | Both values retained                           |
| Correction                | User replaces an earlier date               | Final date used throughout                     |
| Repeated question         | User asks for clarification twice           | Context retained without duplicate action      |
| Unavailable dependency    | Calendar lookup fails                       | Honest fallback; no invented success           |
| Unauthorized action       | User asks to cancel another user’s booking  | Identity and permission boundary respected     |
| Interrupted interaction   | Caller talks over a readback                | Correct recovery and final confirmed values    |
| Repetition or concurrency | Two requests target the same slot           | Controlled collision policy; no double booking |

These are scenario templates. They do not claim that every associated integration or fault-injection mechanism is built into ContextQA. Prepare the environment and verifier needed for each case.

## Review generated coverage

After QuickStart generation, check that each case traces to an actual requirement, that required inputs are present, and that the expected outcome is specific enough to judge. Remove duplicate cases that add no meaningful variation. Keep skipped requirements in a coverage log with an owner and a reason.

Assign [metrics](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/metrics-and-judges.md) that correspond to the assertions. A pronunciation criterion cannot establish booking accuracy. A task-completion rubric based only on the conversation cannot prove persistence.

## Isolate data and identify every run

Give each test a unique customer or correlation identifier when possible. Avoid sharing a single appointment slot between unrelated success cases. Reset test data through an authorized fixture process, or use the appropriate [tool role](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/verify-tool-actions.md). Record intentional collisions as concurrency tests. Never silently reinterpret a fixture conflict as an agent reasoning defect.

Continue with [run comparison](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/runs-and-regression.md) after the cases and their evidence requirements are stable.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/ai-agent-testing/agent-evaluation/design-test-cases.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
