Validate a QA Agent Skill
A reproducible acceptance protocol for testing a versioned QA agent skill, its MCP dependencies, safety boundaries, and evidence before sharing it with a team.
Quick answer
Treat an agent skill like reviewed automation, not a reusable prompt. Pin the exact skill version, run it against a disposable representative fixture, test its expected and failure paths, and retain enough evidence for another reviewer to reproduce the result.
What this page covers
This is a validation method for teams that package QA instructions as reusable agent skills. It does not imply that ContextQA creates or certifies skills for every AI client. The connected client's skill format, permissions, and available ContextQA MCP tools remain authoritative.
Keep the three layers separate
Prompt
The current task and success criteria
Inputs, scope, and requested output
Skill
Versioned workflow instructions and supporting resources
Version, precedence, repeatability, and safety rules
MCP server
Callable ContextQA capabilities
Connected manifest, authentication, permissions, and returned evidence
A skill can tell an agent how to perform a workflow. It cannot create a missing MCP capability, bypass a workspace permission, or make an unsupported action safe.
Create a validation record
Record these fields before the first run:
Skill name and version
qa-release-review at Git commit abc1234
AI client and model
Client name, model, and relevant settings
MCP endpoint and scope
Endpoint name plus connected workspace
Manifest captured at
UTC timestamp and relevant tool names
Fixture
Disposable repository, test plan, and seeded defect
Operator permissions
Role and allowed read, run, change, and release actions
Expected artifacts
Result IDs, screenshots, logs, trace, and decision record
Reviewer
Person independent of the first run where practical
Do not place access tokens, passwords, customer data, or production secrets in the validation record.
Build a representative fixture
Use a user-owned test environment that can be reset. Include:
one stable happy path;
one deterministic application failure;
one locator-only UI change, if self-healing is in scope;
one ambiguous requirement that should make the agent ask or stop;
one unavailable or unauthorized tool request;
known data, tenant, and role boundaries;
a cleanup procedure that returns the fixture to its starting state.
The fixture should resemble the team's work without exposing a live customer or production system.
Run the acceptance protocol
Verify discovery. Confirm the skill version loaded and inspect the connected MCP manifest. Stop if a required capability is absent.
Run a read-only task. Ask the agent to inspect the fixture and propose a plan without creating, editing, or running tests.
Run the happy path. Permit only the minimum actions needed. Reconcile every claimed result to a ContextQA result ID and artifact.
Run the seeded failure. Verify the skill reports the failure instead of rewriting the assertion, healing around the defect, or presenting a pass.
Remove a required capability. Confirm the agent stops, names the missing dependency, and offers a safe handoff.
Reduce permissions. Confirm the agent respects the denied action and does not search for an alternate route around the boundary.
Introduce ambiguity. Confirm the agent requests the missing decision or records an unresolved item rather than inventing it.
Repeat from a clean state. A second operator should be able to follow the record and reach a materially equivalent result.
Score the result
Instruction fidelity
The documented workflow and stop conditions were followed
Transcript excerpt and action log
Capability honesty
Only tools present in the connected manifest were claimed
Manifest snapshot and tool calls
Permission safety
Denied actions remained denied
Role record and error response
Functional detection
The seeded defect failed for the intended reason
Failed assertion and supporting artifacts
Evidence integrity
Every verdict maps to a retrievable result
Result IDs and artifact links
Repeatability
Clean rerun reaches an equivalent verdict
Second-run record
Cleanup
Fixture returns to its documented baseline
Reset verification
Do not average away a failed safety or evidence-integrity gate. Those are blocking failures even if the other scores are strong.
Record limitations and the rollout decision
Classify each finding as:
Accept — suitable for the tested scope;
Revise — change the skill and repeat affected gates;
Restrict — allow only named roles, repositories, or actions;
Stop — do not share until the blocking issue is resolved.
Publish the approved skill version together with its validation record, known limitations, required MCP capabilities, owner, and review date. Any change to those inputs should trigger a targeted revalidation.
Related documentation
Last updated
Was this helpful?