> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-agent-testing/agent-evaluation.md).

# AI Agent Testing with ContextQA

Evaluate AI agents with reviewed scenarios, configurable judges, backend verification, and repeatable evidence in ContextQA.

> **Documentation owner:** [Deep Barot](https://contextqa.com/academy/) · **Last updated:** November 12, 2026

## Quick answer

AI agent testing checks whether an AI system understands a request, follows its rules, uses tools correctly, and produces the required business outcome. In ContextQA’s agent evaluation workspace, connect an agent, create a project, review personas and use cases, generate tests, configure metrics and judges, and inspect run evidence. Evaluate text behavior and voice behavior separately when a workflow supports both.

## Key takeaways

* Test the agent’s decisions, conversation, and business outcomes with explicit acceptance criteria.
* Use backend verification when a successful answer must correspond to a real transaction.
* Keep scenario, agent, judge, and environment versions stable when comparing runs.

## Advanced use cases and the evaluation lifecycle

Use the [capability and use-case map](/ai-agent-testing/agent-evaluation/use-cases.md) for [industry playbooks](/ai-agent-testing/agent-evaluation/industry-use-cases.md), [AI failure modes](/ai-agent-testing/agent-evaluation/failure-modes.md), [pre- and post-deployment testing](/ai-agent-testing/agent-evaluation/deployment-lifecycle.md), [production analytics](/ai-agent-testing/agent-evaluation/production-analytics.md), [agent-builder integration](/ai-agent-testing/agent-evaluation/agent-builders.md), and [post-call outcomes](/ai-agent-testing/agent-evaluation/end-to-end-outcomes.md). Give a coding assistant a [structured evaluation specification](/ai-agent-testing/agent-evaluation/evaluation-specification.md) to preserve the intended scope and evidence requirements.

## Choose the right workflow

| Your task                                                  | Guide                                                                                  |
| ---------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| Evaluate a chatbot, assistant, or tool-using agent         | [First evaluation](/ai-agent-testing/agent-evaluation/quickstart.md)                   |
| Test a telephone or spoken conversation                    | [Voice AI testing](/ai-agent-testing/voice-ai-testing.md)                              |
| Use Claude Code, Cursor, or Codex to test application code | [Testing with AI coding agents](/integrations-and-developer-tools/ai-agent-testing.md) |
| Automate evaluation through an MCP client                  | [Agent evaluation MCP workflow](/ai-agent-testing/agent-evaluation/mcp-workflow.md)    |

The agent evaluation workspace is available at [ai.contextqa.com](https://ai.contextqa.com/dashboard). Its projects, agents, metrics, and runs are separate concepts from browser test cases in the application-testing documentation.

## What you can evaluate

Use a project to organize requirements, personas, use cases, tests, variables, knowledge, tools, metrics, and runs. The current interface exposes Evaluation, Determinism, Red team, and Load run modes. Begin with Evaluation and establish a reliable baseline before expanding the test objective.

| Question                                                    | Evidence to inspect                                                |
| ----------------------------------------------------------- | ------------------------------------------------------------------ |
| Did the agent understand the intent and corrected details?  | Conversation and entity or context criteria                        |
| Was the answer relevant, grounded, and safe?                | Applicable text metrics and judge reasoning                        |
| Did the requested action actually happen?                   | Read-only verifier result correlated to this case                  |
| Did a spoken exchange remain understandable and responsive? | Audio criteria, measured timing, and the recording                 |
| Does a failure reproduce?                                   | Repeated runs using recorded configuration and controlled fixtures |

A conversation score is evidence about the selected criteria. It is not, by itself, proof that a transaction was persisted or that an agent is ready for every production workflow.

## Recommended learning path

1. [Connect your agent](/ai-agent-testing/agent-evaluation/connect-agents.md) and confirm the correct environment.
2. Follow the [first evaluation guide](/ai-agent-testing/agent-evaluation/quickstart.md).
3. [Design scenarios](/ai-agent-testing/agent-evaluation/design-test-cases.md) around business outcomes and failure conditions.
4. Define [metrics and judges](/ai-agent-testing/agent-evaluation/metrics-and-judges.md), then add [tool verification](/ai-agent-testing/agent-evaluation/verify-tool-actions.md).
5. Review [run results and regression evidence](/ai-agent-testing/agent-evaluation/runs-and-regression.md).

## Frequently asked questions

### Is AI agent testing the same as testing AI-generated code?

No. Agent evaluation tests a running AI system’s behavior and outcomes. Testing AI-generated code checks the application code a coding assistant produces. A team may need both workflows.

### Can I use the same project to evaluate text and voice?

The onboarding workflow supports voice and mixed testing. Match each case and metric to its modality. A text result does not establish that speech recognition, turn taking, or audio delivery works.

### Does a successful connection prove the agent works?

A connection establishes access, not task correctness. Run a representative case and inspect its evidence before accepting the integration.

## Scope of this reference

Navigation, connection choices, metric controls, tool roles, and run setup were checked in the live interface on November 11, 2026. Result-reading instructions also use the supplied onboarding guide. Labels and availability can vary by workspace; use the options presented in your environment. The screenshots show configuration screens, not newly executed benchmark results.

[Explore ContextQA](https://contextqa.com/?utm_source=github\&utm_medium=developer_referral\&utm_campaign=gitbooks-docs\&utm_content=agent_evaluation_overview).

## Follow the evaluation modules

1. **Set up:** [Connect an agent](/ai-agent-testing/agent-evaluation/connect-agents.md) and [run a first evaluation](/ai-agent-testing/agent-evaluation/quickstart.md).
2. **Design:** Review [test cases](/ai-agent-testing/agent-evaluation/design-test-cases.md) and [metrics and judges](/ai-agent-testing/agent-evaluation/metrics-and-judges.md).
3. **Verify:** Check [tool actions](/ai-agent-testing/agent-evaluation/verify-tool-actions.md) and [run evidence](/ai-agent-testing/agent-evaluation/runs-and-regression.md).
4. **Expand:** Add [voice testing](/ai-agent-testing/voice-ai-testing.md) or an [MCP workflow](/ai-agent-testing/agent-evaluation/mcp-workflow.md).

Use the [walkthrough library](/walkthroughs-demos-and-examples/learn.md#ai-agent-and-voice-walkthroughs) to watch, try, and verify each stage. For application workflows, return to [the documentation home](/readme.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/ai-agent-testing/agent-evaluation.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.