> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/deployment-lifecycle.md).

# Pre-Deployment and Post-Deployment AI Testing

Plan agent evaluation before release and after deployment with stable baselines, controlled production checks, incident replay, and separate monitoring evidence.

> **Documentation owner:** [Deep Barot](https://contextqa.com/academy/) · **Last updated:** November 12, 2026

## Quick answer

Pre-deployment testing checks a candidate agent against reviewed requirements before release. Post-deployment testing checks the deployed system and turns observed failures into regression cases. ContextQA’s evaluation workflow supports explicit scenarios and evidence review; production telemetry ingestion, schedules, and alerts must be verified separately for the chosen integration.

## Key takeaways

* Preserve the tested agent revision, environment, fixtures, and judge policy.
* Distinguish synthetic checks from passive analysis of real interactions.
* Promote observed incidents into sanitized, reproducible regression cases.

## Choose the lifecycle stage

| Stage                       | Purpose                                                            | Environment and evidence                                           |
| --------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------ |
| Development                 | Reproduce one failure or validate one change                       | Isolated agent revision and controlled case                        |
| Pre-deployment              | Evaluate required behaviors before release                         | Candidate environment, reviewed baseline, critical assertions      |
| Deployment smoke check      | Confirm the released endpoint is reachable and behaves as intended | Controlled synthetic account and a bounded workflow                |
| Post-deployment evaluation  | Detect changed behavior and validate fixes                         | Deployed revision, authorized test scope, stable criteria          |
| Passive production analysis | Inspect real interactions and observed outcomes                    | Authorized telemetry source, sampling policy, privacy controls     |
| Incident-to-regression      | Prevent recurrence of an observed failure                          | Sanitized reproducer and independently established expected result |

The last two stages require an evidence pipeline. Do not assume that starting an Evaluation run automatically imports production calls or configures continuous monitoring.

## Before deployment

1. Define the release scope and critical business invariants with the process owner.
2. Record the agent, model/prompt/configuration, tool schema, knowledge, and environment revisions that affect behavior.
3. Prepare a baseline with normal, negative, correction, escalation, and dependency-failure scenarios.
4. Configure [metrics and judges](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/metrics-and-judges.md) and [backend verification](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/verify-tool-actions.md).
5. Run the selected cases and inspect critical failures individually.
6. Repeat variable cases and evaluate valid semantic alternatives rather than requiring identical wording.
7. Record the release decision, exclusions, and unresolved evidence gaps.

Do not tune every example to the candidate. Include withheld paraphrases or persona variations and independent reference labels. A stricter rubric creates a new evaluation baseline; it is not evidence that the target suddenly became worse.

## After deployment

First check the deployed identity and routing. A passing staging run does not establish that production uses the same model, prompt, knowledge source, permissions, or endpoint. Use controlled test accounts and a bounded scenario. Prefer non-mutating checks where they can establish the requirement; if a transaction must be exercised, explicitly define permitted data, cleanup, and stopping conditions.

Choose a cadence based on the workflow’s risk and supported orchestration. This guide does not provide an unverified scheduler or production-monitoring setup. Verify the run trigger, target selection, result retrieval, alert destination, and failure handling in the actual integration before relying on it.

## Convert an incident into a durable test

Capture the relevant conversation or audio interval and correlated action records. Remove unnecessary personal data, then write the expected behavior independently of the observed answer. Reproduce the issue in a suitable lower environment. Preserve the defect case, propose a targeted change, and rerun the original and neighboring cases after the revised agent is deployed to that environment.

For a post-call failure, keep call end time and downstream completion time separate. A repaired summary generator is not proof that the live voice agent or booking tool was repaired. Follow [end-to-end outcomes](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/end-to-end-outcomes.md).

## Release review record

Use a review table containing the candidate revision, baseline revision, selected cases, critical criteria, failures, inconclusive cases, missing evidence, and approving reviewer. Link each decision to a run/case identifier. Track deployed behavior separately from historical evaluation scores.

Continue with [production analytics](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/production-analytics.md) for measurement definitions and [runs and regression](https://learning.contextqa.com/ai-agent-testing/agent-evaluation/runs-and-regression.md) for result interpretation.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/ai-agent-testing/agent-evaluation/deployment-lifecycle.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
