> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/evaluation-and-governance/ci-integration-acceptance-runbook.md). # CI Integration Acceptance Runbook Test the start, poll, identity, terminal-state, timeout, retry, evidence, and rollback behavior of a ContextQA CI quality gate. ## Quick answer Keep a ContextQA CI job advisory until you have exercised its pass, test-failure, authentication-failure, queue-timeout, execution-timeout, cancellation, retry, stale-result, and wrong-build paths. The adapter must preserve the ContextQA run reference, bind it to the intended commit and environment, map every documented terminal state, and fail closed when the outcome is incomplete or unknown. Promote the job to required only after reviewers can reproduce those behaviors and roll back the policy safely. ## Why a passing test is not enough A CI quality gate coordinates at least three state machines: ```mermaid sequenceDiagram participant CI as CI job participant Adapter as ContextQA adapter participant CQA as ContextQA execution participant App as Candidate environment CI->>Adapter: Start saved plan + expected build identity Adapter->>CQA: Published start operation CQA-->>Adapter: Run reference Adapter->>App: Verify candidate identity loop Bounded polling Adapter->>CQA: Published status operation CQA-->>Adapter: Current state and result end Adapter-->>CI: Pass / Fail / Unknown + run reference ``` A test can be correct while the gate is unsafe. Common examples include polling the wrong run, treating a timeout as success, retrying a non-idempotent start request, losing the run link, or exercising a stale deployment. {% hint style="warning" %} The verified ContextQA **CI / CD** screen describes start and status operations but does not publish endpoint paths in this documentation. Use the public API contract supplied to your workspace. Do not copy internal portal URLs or substitute an invented endpoint. {% endhint %} ## Define the adapter contract Keep vendor-specific HTTP details behind one small adapter. Whether the caller is GitHub Actions, Jenkins, GitLab CI, CircleCI, Buildkite, or another system, the caller should provide the same inputs and receive the same result. ### Inputs ``` test_plan_id Non-secret saved-plan identifier expected_commit Full commit SHA expected_build Immutable artifact or deployment identifier environment Named execution target gate_timeout_seconds Maximum time CI will wait poll_interval_seconds Interval allowed by the public contract correlation_id Unique identifier for this CI attempt ``` The API token and base URL should come from the CI secret/configuration store, not command-line output or a committed file. ### Output ```json { "outcome": "PASS | FAIL | UNKNOWN", "reason": "short machine-readable reason", "run_reference": "value returned by the published contract", "expected_commit": "full SHA", "observed_build": "verified application identity or null", "last_observed_state": "published ContextQA state", "started_at": "timestamp", "finished_at": "timestamp or null", "evidence_reference": "published reference or null" } ``` This schema is a team-owned wrapper, not a representation of the ContextQA API response. Map the official response explicitly inside the adapter. ## Separate execution state from test result Do not collapse these concepts: * **Request state:** was the start call accepted, rejected, or ambiguous? * **Execution state:** is the run queued, active, terminal, canceled, or unavailable? * **Test result:** did the completed test plan pass or fail its assertions? * **Gate outcome:** should CI allow, block, or report an unknown decision? Create a mapping table from the exact values in your public contract. | Published state/result | Terminal? | Gate outcome | Reason | Retry allowed? | | --------------------------------- | --------- | ------------------------- | ------------------ | ---------------- | | Documented success | Yes | Pass | `tests_passed` | No | | Documented test failure | Yes | Fail | `tests_failed` | Policy-specific | | Documented canceled state | Yes | Unknown | `run_canceled` | New attempt only | | Documented infrastructure failure | Yes | Unknown or Fail by policy | `execution_failed` | Bounded | | Queued/running | No | Continue polling | `in_progress` | Not applicable | | Unknown response value | Unknown | Unknown | `unmapped_state` | Stop and alert | | Gate timeout | No result | Unknown | `gate_timeout` | New attempt only | Unknown values must fail closed. Add them to the mapping only after reviewing the updated contract. CI systems also have their own states. For example, GitHub Actions exposes success, failure, canceled, and skipped conclusions, while Buildkite documents states including passed, failed, canceled, skipped, not-run, blocked, and timed-out jobs. Keep the ContextQA-to-CI translation explicit rather than assuming the names align. See the official [GitHub Actions contexts reference](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts) and [Buildkite Builds API](https://buildkite.com/docs/apis/rest-api/builds). ## Bind the gate to the intended build Record: | Identity | Source | | ------------------------------- | ------------------------------------------ | | Candidate commit SHA | CI event payload | | Artifact or image digest | Build system | | Deployment/environment revision | Deployment system | | Application-reported version | Test-only endpoint or visible build marker | | ContextQA test plan | Protected CI variable | | ContextQA run | Start-operation response | Before returning `Pass`, verify that the application-reported or deployment identity matches the candidate. If this is not technically possible, record the limitation and keep the gate advisory for decisions that require candidate-specific proof. ## Make start requests safe The start call can fail in three ways: 1. **Rejected:** the service returns a clear authentication, authorization, validation, or rate-limit error. 2. **Accepted:** the service returns a run reference. 3. **Ambiguous:** the connection fails after the request may have reached the service, but before the caller receives a run reference. Do not automatically repeat an ambiguous start unless the published contract provides an idempotency mechanism or a lookup by your correlation ID. Otherwise, a retry can create duplicate runs. Use a unique correlation ID per CI attempt and log it before the start call. If the official contract supports an idempotency key, bind it to that correlation ID and keep it stable across transport retries. ## Poll with two time budgets Use separate limits: * **Execution budget:** the maximum duration allowed for the saved ContextQA plan. * **Gate budget:** how long the CI worker will wait, including queueing and polling. The gate budget should be longer than the normal execution budget but shorter than the CI system's hard termination limit. Add jitter only if the API guidance permits it; never poll more frequently than documented. Pseudocode: ``` run = start(plan, correlation_id) if run has no stable reference: return UNKNOWN(start_ambiguous_or_rejected) verify candidate build identity if identity mismatches: return UNKNOWN(wrong_build) until gate deadline: state = get_status(run.reference) if state is documented terminal success: return PASS with run.reference if state is documented terminal test failure: return FAIL with run.reference if state is documented terminal incomplete/fault state: return UNKNOWN with run.reference if state is unknown: return UNKNOWN(unmapped_state) wait documented interval return UNKNOWN(gate_timeout) with run.reference and last state ``` Never start a second run merely because polling failed once. Retry safe status reads with bounded exponential backoff, preserving the original run reference. ## Preserve evidence for humans The CI summary should be short enough to scan and specific enough to investigate. ``` ContextQA quality gate: UNKNOWN Reason: gate_timeout Candidate commit: Observed build: Plan: Run: Last state: Elapsed: Next action: inspect the run; do not rerun until duplicate-start risk is resolved ``` Do not synthesize a deep link unless the public response returns or documents one. Link reviewers to [test results](/reporting/test-results.md) using the stable reference available to your workspace. Redact: * tokens and authorization headers; * cookies and session identifiers; * private test data; * request or response bodies containing credentials; and * temporary signed artifact URLs when retention policy forbids exposure. ## Run the acceptance matrix Create a disposable branch and test plan. Do not use a release-blocking production workflow for fault injection. | ID | Scenario | How to induce it safely | Expected outcome | | ----- | ------------------------- | ------------------------------------------------------- | ------------------------------------------------- | | CI-01 | Required assertions pass | Known-good disposable build | Pass | | CI-02 | Required assertion fails | Seed a reversible fixture defect | Fail | | CI-03 | Invalid token | Use an intentionally invalid disposable secret | Unknown with auth reason; no secret in logs | | CI-04 | Insufficient scope | Use a restricted disposable token | Unknown with permission reason | | CI-05 | Start request rejected | Invalid plan ID in disposable config | Unknown; no polling begins | | CI-06 | Ambiguous start | Simulate a transport break after dispatch | Unknown; no unsafe duplicate start | | CI-07 | Queue exceeds gate budget | Use a test-only short gate timeout | Unknown with last observed state | | CI-08 | Test execution times out | Controlled slow fixture | Unknown or policy-defined non-success, never Pass | | CI-09 | Run canceled | Cancel the disposable run | Unknown with cancellation preserved | | CI-10 | Status read briefly fails | Inject a transient read failure | Original run is polled after bounded retry | | CI-11 | Unknown response state | Adapter unit fixture | Unknown and alert; no implicit success | | CI-12 | Wrong application build | Point expected identity at a different disposable build | Unknown before functional pass is accepted | | CI-13 | Evidence unavailable | Restrict reviewer access in a test context | Unknown or explicit access failure | | CI-14 | CI job is canceled | Cancel the disposable CI attempt | Cleanup runs without converting result to Pass | | CI-15 | Retry after test failure | Start a new authorized attempt | New run reference; original evidence remains | For GitHub Actions, a nonzero process exit code produces a failing check. The official guidance also notes that a failed action affects concurrent and future actions, so test your cleanup and summary steps under failure rather than assuming they run. See [Setting exit codes for actions](https://docs.github.com/en/actions/how-tos/create-and-publish-actions/set-exit-codes). ## Verify retry policy Define retry by failure class: | Failure class | Same request retry? | New execution? | Human approval? | | ------------------------ | ------------------------------------- | --------------------- | ------------------------------- | | Transient status read | Yes, bounded | No | No | | Rate limited status read | Per official backoff guidance | No | No | | Ambiguous start | Only with verified idempotency/lookup | No by default | Yes if unresolved | | Test assertion failure | No | Policy-specific rerun | Often required for release gate | | Environment unavailable | No | After restoration | Owner required | | Invalid credentials | No | No | Credential owner required | | Unknown state mapping | No | No | Adapter owner required | Retries must not erase the first attempt. A later pass can be useful, but the evidence packet should retain the original failure and explain why a retry was allowed. ## Assign ownership | Boundary | Owner | | ------------------------------------------- | --------------------------- | | ContextQA plan contents and assertions | QA or test owner | | Token creation, expiry, and least privilege | ContextQA administrator | | CI secret and variable configuration | Repository/CI administrator | | Adapter implementation and state mapping | Automation owner | | Candidate environment and build identity | Release engineering | | Evidence review and release disposition | Named release owner | | API contract changes | Integration owner | Avoid “DevOps” or “QA” as the only owner. Name a person or an accountable team with an escalation path. ## Promotion checklist * [ ] The saved ContextQA plan passed manually against the intended environment. * [ ] The adapter uses only the public contract supplied to the workspace. * [ ] Secrets are protected and never printed. * [ ] Start ambiguity and duplicate-run behavior are understood. * [ ] Every published execution state has an explicit gate mapping. * [ ] Unknown states fail closed and alert the integration owner. * [ ] Candidate commit/build identity is verified before `Pass`. * [ ] CI-01 through CI-15 have recorded results or approved, documented exceptions. * [ ] Reviewers can reach the run evidence without privileged developer tooling. * [ ] Required-check name and workflow ownership are stable. * [ ] The branch/ruleset change has a rollback owner and procedure. * [ ] The gate ran advisory for the agreed observation window. Promote the gate through normal repository review. Do not change branch protection as an incidental implementation step. ## Rollback plan Before making the check required, record: ``` Required check name: Ruleset or branch policy owner: Prior policy: Rollback condition: Authorized rollback operator: Maximum decision time: Fallback verification: Incident evidence location: ``` Rollback the enforcement—not the evidence—when the adapter is unhealthy. Keep ContextQA runs and CI logs needed to diagnose the incident. ## Common questions
Can we publish a ready-to-run Buildkite or GitHub workflow now? Only after the exact ContextQA public start/status paths, authentication scheme, payload, response fields, and terminal states for your workspace are available. Until then, use this acceptance contract and keep the platform-specific step as a reviewed adapter placeholder.
Should a timed-out run fail the build? A required gate should block, but its reason should remain `Unknown: timeout` rather than `tests failed`. That distinction preserves accurate reliability data and routes the issue to the right owner.
When can a retry turn the gate green? Only under a documented retry policy. Preserve both run references, classify the original failure, and prevent a rerun from hiding an assertion failure or wrong-build result.
## Related documentation * [Evaluation and governance](/evaluation-and-governance/evaluation.md) * [Prepare a CI/CD quality gate](/getting-started/ci-cd-tutorial.md) * [GitHub Actions preparation](/integrations/github-actions.md) * [API tokens](/administration/api-tokens.md) * [Test results](/reporting/test-results.md) {% hint style="info" %} **Need to prove a CI gate before enforcing it?** [**Book an integration review →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=ci_acceptance_runbook) — Bring your CI system, saved plan, and public API contract; we will walk through the negative-path matrix with your team. {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://learning.contextqa.com/evaluation-and-governance/ci-integration-acceptance-runbook.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.