CI Integration Acceptance Runbook

Test the start, poll, identity, terminal-state, timeout, retry, evidence, and rollback behavior of a ContextQA CI quality gate.

Quick answer

Keep a ContextQA CI job advisory until you have exercised its pass, test-failure, authentication-failure, queue-timeout, execution-timeout, cancellation, retry, stale-result, and wrong-build paths. The adapter must preserve the ContextQA run reference, bind it to the intended commit and environment, map every documented terminal state, and fail closed when the outcome is incomplete or unknown. Promote the job to required only after reviewers can reproduce those behaviors and roll back the policy safely.

Why a passing test is not enough

A CI quality gate coordinates at least three state machines:

A test can be correct while the gate is unsafe. Common examples include polling the wrong run, treating a timeout as success, retrying a non-idempotent start request, losing the run link, or exercising a stale deployment.

The verified ContextQA CI / CD screen describes start and status operations but does not publish endpoint paths in this documentation. Use the public API contract supplied to your workspace. Do not copy internal portal URLs or substitute an invented endpoint.

Define the adapter contract

Keep vendor-specific HTTP details behind one small adapter. Whether the caller is GitHub Actions, Jenkins, GitLab CI, CircleCI, Buildkite, or another system, the caller should provide the same inputs and receive the same result.

Inputs

test_plan_id          Non-secret saved-plan identifier
expected_commit       Full commit SHA
expected_build        Immutable artifact or deployment identifier
environment           Named execution target
gate_timeout_seconds  Maximum time CI will wait
poll_interval_seconds Interval allowed by the public contract
correlation_id        Unique identifier for this CI attempt

The API token and base URL should come from the CI secret/configuration store, not command-line output or a committed file.

Output

{
  "outcome": "PASS | FAIL | UNKNOWN",
  "reason": "short machine-readable reason",
  "run_reference": "value returned by the published contract",
  "expected_commit": "full SHA",
  "observed_build": "verified application identity or null",
  "last_observed_state": "published ContextQA state",
  "started_at": "timestamp",
  "finished_at": "timestamp or null",
  "evidence_reference": "published reference or null"
}

This schema is a team-owned wrapper, not a representation of the ContextQA API response. Map the official response explicitly inside the adapter.

Separate execution state from test result

Do not collapse these concepts:

• Request state: was the start call accepted, rejected, or ambiguous?
• Execution state: is the run queued, active, terminal, canceled, or unavailable?
• Test result: did the completed test plan pass or fail its assertions?
• Gate outcome: should CI allow, block, or report an unknown decision?

Create a mapping table from the exact values in your public contract.

Published state/result	Terminal?	Gate outcome	Reason	Retry allowed?
Documented success	Yes	Pass	tests_passed	No
Documented test failure	Yes	Fail	tests_failed	Policy-specific
Documented canceled state	Yes	Unknown	run_canceled	New attempt only
Documented infrastructure failure	Yes	Unknown or Fail by policy	execution_failed	Bounded
Queued/running	No	Continue polling	in_progress	Not applicable
Unknown response value	Unknown	Unknown	unmapped_state	Stop and alert
Gate timeout	No result	Unknown	gate_timeout	New attempt only

Unknown values must fail closed. Add them to the mapping only after reviewing the updated contract.

CI systems also have their own states. For example, GitHub Actions exposes success, failure, canceled, and skipped conclusions, while Buildkite documents states including passed, failed, canceled, skipped, not-run, blocked, and timed-out jobs. Keep the ContextQA-to-CI translation explicit rather than assuming the names align. See the official GitHub Actions contexts reference and Buildkite Builds API.

Bind the gate to the intended build

Record:

Identity	Source
Candidate commit SHA	CI event payload
Artifact or image digest	Build system
Deployment/environment revision	Deployment system
Application-reported version	Test-only endpoint or visible build marker
ContextQA test plan	Protected CI variable
ContextQA run	Start-operation response

Before returning Pass, verify that the application-reported or deployment identity matches the candidate. If this is not technically possible, record the limitation and keep the gate advisory for decisions that require candidate-specific proof.

Make start requests safe

The start call can fail in three ways:

1. Rejected: the service returns a clear authentication, authorization, validation, or rate-limit error.
2. Accepted: the service returns a run reference.
3. Ambiguous: the connection fails after the request may have reached the service, but before the caller receives a run reference.

Do not automatically repeat an ambiguous start unless the published contract provides an idempotency mechanism or a lookup by your correlation ID. Otherwise, a retry can create duplicate runs.

Use a unique correlation ID per CI attempt and log it before the start call. If the official contract supports an idempotency key, bind it to that correlation ID and keep it stable across transport retries.

Poll with two time budgets

Use separate limits:

• Execution budget: the maximum duration allowed for the saved ContextQA plan.
• Gate budget: how long the CI worker will wait, including queueing and polling.

The gate budget should be longer than the normal execution budget but shorter than the CI system's hard termination limit. Add jitter only if the API guidance permits it; never poll more frequently than documented.

Pseudocode:

run = start(plan, correlation_id)
if run has no stable reference:
    return UNKNOWN(start_ambiguous_or_rejected)

verify candidate build identity
if identity mismatches:
    return UNKNOWN(wrong_build)

until gate deadline:
    state = get_status(run.reference)
    if state is documented terminal success:
        return PASS with run.reference
    if state is documented terminal test failure:
        return FAIL with run.reference
    if state is documented terminal incomplete/fault state:
        return UNKNOWN with run.reference
    if state is unknown:
        return UNKNOWN(unmapped_state)
    wait documented interval

return UNKNOWN(gate_timeout) with run.reference and last state

Never start a second run merely because polling failed once. Retry safe status reads with bounded exponential backoff, preserving the original run reference.

Preserve evidence for humans

The CI summary should be short enough to scan and specific enough to investigate.

ContextQA quality gate: UNKNOWN
Reason: gate_timeout
Candidate commit: <full SHA>
Observed build: <build ID or unavailable>
Plan: <plan reference>
Run: <run reference>
Last state: <state>
Elapsed: <duration>
Next action: inspect the run; do not rerun until duplicate-start risk is resolved

Do not synthesize a deep link unless the public response returns or documents one. Link reviewers to test results using the stable reference available to your workspace.

Redact:

• tokens and authorization headers;
• cookies and session identifiers;
• private test data;
• request or response bodies containing credentials; and
• temporary signed artifact URLs when retention policy forbids exposure.

Run the acceptance matrix

Create a disposable branch and test plan. Do not use a release-blocking production workflow for fault injection.

ID	Scenario	How to induce it safely	Expected outcome
CI-01	Required assertions pass	Known-good disposable build	Pass
CI-02	Required assertion fails	Seed a reversible fixture defect	Fail
CI-03	Invalid token	Use an intentionally invalid disposable secret	Unknown with auth reason; no secret in logs
CI-04	Insufficient scope	Use a restricted disposable token	Unknown with permission reason
CI-05	Start request rejected	Invalid plan ID in disposable config	Unknown; no polling begins
CI-06	Ambiguous start	Simulate a transport break after dispatch	Unknown; no unsafe duplicate start
CI-07	Queue exceeds gate budget	Use a test-only short gate timeout	Unknown with last observed state
CI-08	Test execution times out	Controlled slow fixture	Unknown or policy-defined non-success, never Pass
CI-09	Run canceled	Cancel the disposable run	Unknown with cancellation preserved
CI-10	Status read briefly fails	Inject a transient read failure	Original run is polled after bounded retry
CI-11	Unknown response state	Adapter unit fixture	Unknown and alert; no implicit success
CI-12	Wrong application build	Point expected identity at a different disposable build	Unknown before functional pass is accepted
CI-13	Evidence unavailable	Restrict reviewer access in a test context	Unknown or explicit access failure
CI-14	CI job is canceled	Cancel the disposable CI attempt	Cleanup runs without converting result to Pass
CI-15	Retry after test failure	Start a new authorized attempt	New run reference; original evidence remains

For GitHub Actions, a nonzero process exit code produces a failing check. The official guidance also notes that a failed action affects concurrent and future actions, so test your cleanup and summary steps under failure rather than assuming they run. See Setting exit codes for actions.

Verify retry policy

Define retry by failure class:

Failure class	Same request retry?	New execution?	Human approval?
Transient status read	Yes, bounded	No	No
Rate limited status read	Per official backoff guidance	No	No
Ambiguous start	Only with verified idempotency/lookup	No by default	Yes if unresolved
Test assertion failure	No	Policy-specific rerun	Often required for release gate
Environment unavailable	No	After restoration	Owner required
Invalid credentials	No	No	Credential owner required
Unknown state mapping	No	No	Adapter owner required

Retries must not erase the first attempt. A later pass can be useful, but the evidence packet should retain the original failure and explain why a retry was allowed.

Assign ownership

Boundary	Owner
ContextQA plan contents and assertions	QA or test owner
Token creation, expiry, and least privilege	ContextQA administrator
CI secret and variable configuration	Repository/CI administrator
Adapter implementation and state mapping	Automation owner
Candidate environment and build identity	Release engineering
Evidence review and release disposition	Named release owner
API contract changes	Integration owner

Avoid “DevOps” or “QA” as the only owner. Name a person or an accountable team with an escalation path.

Promotion checklist

• The saved ContextQA plan passed manually against the intended environment.
• The adapter uses only the public contract supplied to the workspace.
• Secrets are protected and never printed.
• Start ambiguity and duplicate-run behavior are understood.
• Every published execution state has an explicit gate mapping.
• Unknown states fail closed and alert the integration owner.
• Candidate commit/build identity is verified before Pass.
• CI-01 through CI-15 have recorded results or approved, documented exceptions.
• Reviewers can reach the run evidence without privileged developer tooling.
• Required-check name and workflow ownership are stable.
• The branch/ruleset change has a rollback owner and procedure.
• The gate ran advisory for the agreed observation window.

Promote the gate through normal repository review. Do not change branch protection as an incidental implementation step.

Rollback plan

Before making the check required, record:

Required check name:
Ruleset or branch policy owner:
Prior policy:
Rollback condition:
Authorized rollback operator:
Maximum decision time:
Fallback verification:
Incident evidence location:

Rollback the enforcement—not the evidence—when the adapter is unhealthy. Keep ContextQA runs and CI logs needed to diagnose the incident.

Common questions

Can we publish a ready-to-run Buildkite or GitHub workflow now?

Only after the exact ContextQA public start/status paths, authentication scheme, payload, response fields, and terminal states for your workspace are available. Until then, use this acceptance contract and keep the platform-specific step as a reviewed adapter placeholder.

Should a timed-out run fail the build?

A required gate should block, but its reason should remain Unknown: timeout rather than tests failed. That distinction preserves accurate reliability data and routes the issue to the right owner.

When can a retry turn the gate green?

Only under a documented retry policy. Preserve both run references, classify the original failure, and prevent a rerun from hiding an assertion failure or wrong-build result.

Related documentation

• Evaluation and governance
• Prepare a CI/CD quality gate
• GitHub Actions preparation
• API tokens
• Test results

Need to prove a CI gate before enforcing it? Book an integration review → — Bring your CI system, saved plan, and public API contract; we will walk through the negative-path matrix with your team.