> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/metrics-and-judges.md).

# Configure AI Agent Metrics and Judges

Configure deterministic checks, custom LLM rubrics, metric scope, and text or audio judges for evidence-based AI agent evaluation.

> **Documentation owner:** [Deep Barot](https://contextqa.com/academy/) · **Last updated:** November 11, 2026

## Quick answer

ContextQA scores agent behavior using deterministic criteria and LLM-judged metrics. Select metrics that match the workflow, define explicit pass, partial, and fail conditions, and configure the run’s text and audio judges. Treat weights and pass thresholds as evaluation settings to calibrate against reviewed cases, not as universal measures of agent quality.

## Key takeaways

* Use exact checks for exact requirements and semantic rubrics for contextual judgment.
* Separate text and audio evaluation and inspect the reasoning behind each score.
* Use tools or knowledge only when the criterion requires that evidence.

## Start with the available criteria

The checked project offered text criteria including Intent Recognition, Entity Extraction, Task Completion, Hallucination, Groundedness, Response Relevance, Safety / Toxicity, Escalation Accuracy, Response Latency, Multi-Turn Context, Fallback Handling, and KB Answer Accuracy. The interface identifies which are deterministic, LLM judged, or associated with tools or knowledge.

Select criteria according to the requirement. A knowledge-grounding test needs the relevant source content. An action-completion test needs evidence of the action. For spoken interactions, also review [voice metrics](https://learning.contextqa.com/ai-agent-testing/voice-ai-testing/metrics-reference.md).

## Create a custom metric

1. Open the project’s **Metrics** page.
2. Choose **New Custom Metric → Add Manually**, or use **Generate with Copilot** and review its proposal.
3. Set the name, input modality, description, and weight.
4. Review the **Applies to** scope. Use every-case scope only for a requirement that really applies to every case.
5. Choose **LLM Judge** or **Deterministic** as appropriate.
6. For an LLM metric, write explicit PASS, PARTIAL, and FAIL conditions.
7. Enable tool or knowledge verification when the rubric requires it, and verify that the supporting resources are available.

![Custom metric form showing modality, weight, scope, judgement type, and rubric controls](https://4255556216-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKlyaXxb76z3Dn47TrZkU%2Fuploads%2Fgit-blob-2700eb7ca0673ba6622bd05ef0b9f6757d0e3c2a%2Fcustom-metric.jpg?alt=media)

## Example rubric: final booking confirmation

| Outcome | Rule                                                                                                                                         |
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| PASS    | The agent reads back the final corrected service, date, time, and timezone and receives affirmative confirmation before creating the booking |
| PARTIAL | The agent asks for confirmation but omits a required noncritical detail; no contradictory booking is created                                 |
| FAIL    | The agent uses superseded values, treats a rejection as confirmation, or claims completion with contradictory evidence                       |

Decide which omissions are critical before running the evaluation. If a date, timezone, or identity error is unacceptable, make it a failure rather than allowing an average score to hide it. Use a separate [backend verification criterion](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/verify-tool-actions.md) to establish that the persisted booking matches.

Write atomic criteria where possible. Separate name accuracy from date accuracy so a corrected date cannot conceal a remaining name error. Include positive and negative examples to calibrate ambiguous rubric language.

## Configure judges and thresholds

On **New Run**, select a judge preset or override it with **Custom judge configuration**. The current interface presents text and audio judges separately, along with pass thresholds. Model options change; use those listed in your workspace and review the displayed cost estimate.

![Run configuration drawer showing separate text and audio judges and pass thresholds](https://4255556216-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKlyaXxb76z3Dn47TrZkU%2Fuploads%2Fgit-blob-00f9bd397a9d079c5b72db45cf1957d54729aa05%2Frun-judges.jpg?alt=media)

The threshold values shown in this screenshot are example interface values, not recommended release thresholds. Choose a policy based on known good and known bad cases, then keep that policy stable across the comparison.

A project preset applies unless the run overrides it. If no project preset is selected, the workspace default is used. A run needs a valid configuration to start. Record the selected judges and rubric revision with the run evidence.

## Calibrate the judges against independent evidence

Prepare a small reference set labeled by a person who understands the requirement. Include known failures, valid alternative answers, ambiguous cases, and cases with missing evidence. Keep some examples out of the generation and tuning loop so the evaluation is not merely rewarding scenarios optimized for its own rubric.

Compare each judge’s decisions with the reviewed labels. Investigate false passes as well as false failures, especially on critical transaction requirements. Use deterministic checks for exact identifiers, required fields, and backend facts where applicable. Multiple LLM judges can still share blind spots; agreement is not independent proof of correctness.

A score or pass threshold is an evaluation policy, not a statistical confidence level or a probability that the agent is safe. Document why the selected policy is appropriate, who reviews disputes, and which failures block release regardless of the aggregate. Recheck calibration when the judge model, rubric, source knowledge, or target workflow changes.

## Interpret weights and disagreements

Weights express relative importance among applicable metrics; do not infer an undocumented scoring formula from the displayed default. Review critical failures directly rather than relying exclusively on an aggregate.

If judges disagree, compare their evidence and rubric interpretations. Check for missing context, ambiguous acceptance criteria, or a modality mismatch. Human review of a disputed case should document the reason for the decision. Do not quietly change a rubric and compare its new score with the old baseline as though the evaluation were unchanged.

Continue with [runs and regression](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/runs-and-regression.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/ai-agent-testing/agent-evaluation/metrics-and-judges.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
