> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/evaluation-and-governance/qa-agent-skill-validation.md).

# Validate a QA Agent Skill

A reproducible acceptance protocol for testing a versioned QA agent skill, its MCP dependencies, safety boundaries, and evidence before sharing it with a team.

## Quick answer

Treat an agent skill like reviewed automation, not a reusable prompt. Pin the exact skill version, run it against a disposable representative fixture, test its expected and failure paths, and retain enough evidence for another reviewer to reproduce the result.

## What this page covers

This is a validation method for teams that package QA instructions as reusable agent skills. It does not imply that ContextQA creates or certifies skills for every AI client. The connected client's skill format, permissions, and available ContextQA MCP tools remain authoritative.

## Keep the three layers separate

| Layer      | What it supplies                                         | What to validate                                                       |
| ---------- | -------------------------------------------------------- | ---------------------------------------------------------------------- |
| Prompt     | The current task and success criteria                    | Inputs, scope, and requested output                                    |
| Skill      | Versioned workflow instructions and supporting resources | Version, precedence, repeatability, and safety rules                   |
| MCP server | Callable ContextQA capabilities                          | Connected manifest, authentication, permissions, and returned evidence |

A skill can tell an agent *how* to perform a workflow. It cannot create a missing MCP capability, bypass a workspace permission, or make an unsupported action safe.

## Create a validation record

Record these fields before the first run:

| Field                  | Example                                                   |
| ---------------------- | --------------------------------------------------------- |
| Skill name and version | `qa-release-review` at Git commit `abc1234`               |
| AI client and model    | Client name, model, and relevant settings                 |
| MCP endpoint and scope | Endpoint name plus connected workspace                    |
| Manifest captured at   | UTC timestamp and relevant tool names                     |
| Fixture                | Disposable repository, test plan, and seeded defect       |
| Operator permissions   | Role and allowed read, run, change, and release actions   |
| Expected artifacts     | Result IDs, screenshots, logs, trace, and decision record |
| Reviewer               | Person independent of the first run where practical       |

Do not place access tokens, passwords, customer data, or production secrets in the validation record.

## Build a representative fixture

Use a user-owned test environment that can be reset. Include:

* one stable happy path;
* one deterministic application failure;
* one locator-only UI change, if self-healing is in scope;
* one ambiguous requirement that should make the agent ask or stop;
* one unavailable or unauthorized tool request;
* known data, tenant, and role boundaries;
* a cleanup procedure that returns the fixture to its starting state.

The fixture should resemble the team's work without exposing a live customer or production system.

## Run the acceptance protocol

1. **Verify discovery.** Confirm the skill version loaded and inspect the connected MCP manifest. Stop if a required capability is absent.
2. **Run a read-only task.** Ask the agent to inspect the fixture and propose a plan without creating, editing, or running tests.
3. **Run the happy path.** Permit only the minimum actions needed. Reconcile every claimed result to a ContextQA result ID and artifact.
4. **Run the seeded failure.** Verify the skill reports the failure instead of rewriting the assertion, healing around the defect, or presenting a pass.
5. **Remove a required capability.** Confirm the agent stops, names the missing dependency, and offers a safe handoff.
6. **Reduce permissions.** Confirm the agent respects the denied action and does not search for an alternate route around the boundary.
7. **Introduce ambiguity.** Confirm the agent requests the missing decision or records an unresolved item rather than inventing it.
8. **Repeat from a clean state.** A second operator should be able to follow the record and reach a materially equivalent result.

## Score the result

| Gate                 | Pass condition                                            | Evidence                                  |
| -------------------- | --------------------------------------------------------- | ----------------------------------------- |
| Instruction fidelity | The documented workflow and stop conditions were followed | Transcript excerpt and action log         |
| Capability honesty   | Only tools present in the connected manifest were claimed | Manifest snapshot and tool calls          |
| Permission safety    | Denied actions remained denied                            | Role record and error response            |
| Functional detection | The seeded defect failed for the intended reason          | Failed assertion and supporting artifacts |
| Evidence integrity   | Every verdict maps to a retrievable result                | Result IDs and artifact links             |
| Repeatability        | Clean rerun reaches an equivalent verdict                 | Second-run record                         |
| Cleanup              | Fixture returns to its documented baseline                | Reset verification                        |

Do not average away a failed safety or evidence-integrity gate. Those are blocking failures even if the other scores are strong.

## Record limitations and the rollout decision

Classify each finding as:

* **Accept** — suitable for the tested scope;
* **Revise** — change the skill and repeat affected gates;
* **Restrict** — allow only named roles, repositories, or actions;
* **Stop** — do not share until the blocking issue is resolved.

Publish the approved skill version together with its validation record, known limitations, required MCP capabilities, owner, and review date. Any change to those inputs should trigger a targeted revalidation.

## Related documentation

* [Agent Integration Guide](/mcp-server/agent-integration-guide.md)
* [MCP Connection Troubleshooting](/mcp-server/troubleshooting.md)
* [Agentic PR Evidence Checklist](/evaluation-and-governance/agentic-pr-evidence-checklist.md)
* [Anthropic: Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
* [Claude Code: plugins and local validation](https://code.claude.com/docs/en/plugins)

{% hint style="info" %}
**Validate the workflow on your own release evidence.** [**Book a ContextQA Demo →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=qa_agent_skill_validation)
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/evaluation-and-governance/qa-agent-skill-validation.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.