Smallest Operable AI Testing Stack

Compare repository-led, managed-platform, and mixed AI testing architectures by system of record, trigger, execution, evidence, maintenance, and ownership.

Editorial owner: ContextQA Documentation · Last updated: September 12, 2026

Quick answer

The smallest useful stack has one system of record for each test definition, one clear execution path per release gate, one evidence destination reviewers can reach, and a named owner at every integration boundary. Add orchestration, agents, or duplicate runners only when a measured requirement cannot be met more simply.

Start with required capabilities

List the applications, risks, browsers/devices, private-network needs, data constraints, CI triggers, evidence, retention, and portability requirements. Do not start by drawing every available tool.

Pattern 1: repository-led

Choose this when code ownership, framework customization, and direct CI control are central. Your team owns fixtures, dependencies, runner capacity, maintenance, evidence hosting, and failure triage.

Pattern 2: managed platform

Choose this when managed execution and intent-oriented authoring can reduce a measured burden. Your team still owns test oracles, data, environments, access, triage decisions, and release authority.

Pattern 3: mixed

Choose this when different layers need different tools—for example, component and protocol checks in Git, while high-change end-to-end workflows use a managed platform. Prevent duplicate coverage and conflicting systems of record.

Complete the architecture record

Layer	System of record	Trigger	Execution owner	Evidence destination	Failure/maintenance owner
Requirements
Component/API tests
End-to-end tests
Specialist tests
Environments/data
Release decision

For each connection, record authentication, scope, retry behavior, timeout, revocation, observability, and who is paged when it fails.

Remove unnecessary layers

Challenge every component:

• Does it own unique coverage or duplicate another path?
• Is it the source of truth or only a copy?
• Does it improve a measured outcome?
• Can the team diagnose and upgrade it?
• What happens when it is unavailable?
• Can the workflow exit without it?

Avoid a workflow graph when a bounded sequence is sufficient. Avoid two reporting systems when one can carry the required evidence. Avoid model routing when direct provider access already meets policy and reliability needs.

Acceptance test the stack

Run one normal case, one seeded product failure, one unavailable dependency, one timeout, and one revoked-credential scenario. Confirm the correct owner receives an actionable, non-duplicated signal and the release outcome remains pass, fail, or unknown—not silently green.

Frequently asked questions

Does the smallest stack always use the fewest tools?

No. It uses the fewest independently owned layers needed to meet the requirements. A specialist accessibility, performance, or protocol tool can reduce risk; an overlapping runner or duplicate evidence store without a distinct owner usually adds complexity.

When is a mixed testing stack appropriate?

Use a mixed model when different test layers have genuinely different execution, evidence, or ownership requirements. Define one system of record per test, prevent duplicate release signals, and identify how results become one reviewable verdict.

What should be the system of record for tests?

Choose the location where the authoritative test intent, history, ownership, and review occur. Exported or synchronized copies should identify their source and must not become independently edited duplicates without an explicit reconciliation process.

Related documentation

• Anthropic: Building Effective Agents
• Platform Architecture
• Agent Integration Guide
• QA Agent Workflow Graph or Simple Loop

Map the smallest ContextQA pilot architecture → using your current systems of record and ownership boundaries.