> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/production-analytics.md).

# AI Agent Production Analytics and Real-Time Monitoring

Define production analytics for AI agents with explicit ingestion, latency, sampling, outcome, and alert contracts without confusing run reports with continuous monitoring.

> **Documentation owner:** [Deep Barot](https://contextqa.com/academy/) · **Last updated:** November 12, 2026

## Quick answer

Production analytics measures how deployed AI agents behave across real or synthetic interactions. Real-time monitoring additionally requires a verified ingestion path, bounded processing delay, and operational alerts. ContextQA run results provide evaluation evidence; these guides do not establish a universally available streaming-monitoring setup. Validate the deployment’s actual telemetry and alert integration before relying on it.

## Key takeaways

* Run results, production telemetry, and live alerts are different evidence sources.
* Measure business outcomes alongside latency, voice quality, cost, and coverage.
* State sampling, missing data, and measurement delay with every production report.

## What must be configured?

The evaluation workspace exposes run analysis and reporting workflows. Continuous production ingestion and streaming alerts require separate verification. Earlier product discussions described observability work in progress, while later briefings discussed broader use cases. Those discussions alone do not establish a current connector contract, retention policy, refresh interval, or alert SLA.

Use the following acceptance checklist with the team implementing the integration:

| Contract   | Questions to resolve                                                          |
| ---------- | ----------------------------------------------------------------------------- |
| Source     | Which system supplies conversations, audio, tool events, and outcomes?        |
| Identity   | How are tenant, agent revision, session, call, and transaction correlated?    |
| Timing     | Are event time and ingestion time separate? How late can events arrive?       |
| Coverage   | Are all interactions collected, or a sample? What is excluded?                |
| Evaluation | Which rubric/judge revision scores an interaction, and when?                  |
| Security   | Who may access raw content, and what is redacted or retained?                 |
| Alerting   | Which destination is configured, who owns it, and how are duplicates handled? |
| Recovery   | What happens when ingestion, scoring, or the alert destination fails?         |

Treat this as an integration pattern, not a list of controls claimed to be present in every workspace.

## Define a useful measurement set

| Measure                    | Definition to agree on                                                                                | Interpretation trap                                  |
| -------------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| Verified task completion   | Interactions with confirmed required backend outcome / eligible interactions with sufficient evidence | Counting a spoken success as a completed transaction |
| Critical failure rate      | Interactions violating a named critical rule / evaluated eligible interactions                        | Hiding a severe failure inside an average score      |
| Escalation completion      | Required escalations with a verified handoff / interactions requiring escalation                      | Treating “I will transfer you” as a handoff          |
| Response latency           | Defined response interval, reported as a distribution                                                 | Comparing different start/end definitions            |
| Dead air and interruptions | Measures from the relevant audio evidence and criterion definition                                    | Inferring audio behavior from text alone             |
| Outcome cost               | Attributable cost for the workflow and its verified outcomes                                          | Presenting partial token cost as full business cost  |
| Evidence coverage          | Interactions with required evidence / intended eligible interactions                                  | Reporting only successfully scored interactions      |
| Time to detection          | Detection time minus incident time                                                                    | Calling delayed batch analysis real time             |

These definitions are reporting recommendations. They are not claims that each measure is already a native dashboard widget. Preserve failed and inconclusive counts separately and disclose denominators.

## Compare meaningful groups

Segment by agent version, use case, language where available, channel, environment, and relevant deployment cohort. Compare like-for-like populations before concluding that a model change caused a trend. A different mix of difficult calls can change the aggregate without a regression in the agent.

If a new rubric or judge changes scores, separate that evaluation change from the target deployment change. Do not mix synthetic test results and real-user outcomes into an unlabeled success rate.

## Validate an alert before relying on it

Create an authorized, controlled event with a known expected result. Confirm collection, correlation, evaluation, routing, and ownership. Measure elapsed detection time. Also test missing events, duplicate events, scorer unavailability, and alert-delivery failure. An absence of alerts is not evidence of an absence of failures when the pipeline is unhealthy.

Use thresholds and response procedures agreed with the workflow owner. No universal failure-count, delay, or escalation policy is implied here.

## Close the loop

Turn a confirmed incident into a reviewed [regression case](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/deployment-lifecycle.md#convert-an-incident-into-a-durable-test), then compare the fixed candidate against the baseline. Preserve the link between incident, test, change, run, and reviewer decision. See [evaluation specifications](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/evaluation-specification.md) for a portable result record.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/ai-agent-testing/agent-evaluation/production-analytics.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
