GitLab CI

GitLab CI ContextQA integration — trigger test plans from .gitlab-ci.yml, poll for results, and fail pipeline stages automatically when tests fail.

Who is this for? SDETs, developers, and engineering managers who use GitLab CI and want to trigger ContextQA test plans from .gitlab-ci.yml and fail pipeline stages on test failures.

GitLab CI ContextQA integration: A CI/CD configuration where a .gitlab-ci.yml pipeline stage triggers a ContextQA test plan, polls for execution completion, and maps the test result to the GitLab job exit code — blocking merges and deployments when tests fail.

GitLab CI pipelines can call external services between build stages. ContextQA exposes a REST API that GitLab CI jobs can use to trigger a test plan, wait for results, and surface failures as a failed pipeline stage. This page provides a complete .gitlab-ci.yml example and covers credential storage, polling logic, and result publication.

Prerequisites

Before configuring the GitLab CI integration:

• A ContextQA account with at least one configured test plan

• The test plan ID (visible in the URL when viewing the plan: /test-plans/<plan_id>)

• A ContextQA service account (dedicated email and password — do not use a personal account)

• GitLab CI/CD configured on your repository

Storing credentials as GitLab CI/CD variables

Store ContextQA credentials as masked CI/CD variables so they are never visible in job logs.

1. In your GitLab project, navigate to Settings → CI/CD → Variables.

2. Click Add variable.

3. Set Key to CONTEXTQA_USERNAME, Value to your service account email. Check Masked and Protected if your pipelines run on protected branches.

4. Repeat for Key CONTEXTQA_PASSWORD with your service account password.

5. Click Save variables.

GitLab will mask these values in all job logs. The Masked flag prevents the values from appearing even if a job explicitly echoes them.

How the ContextQA API works from GitLab CI

Authentication: POST credentials to the ContextQA login endpoint to receive a Bearer token.

Test plan execution: Send a GET request (not POST) to the execute endpoint. ContextQA returns an execution ID immediately.

Polling: Repeatedly GET the execution status endpoint until the status is PASSED, FAILED, or PARTIAL.

The base URL for all API calls is https://server.contextqa.com.

Complete .gitlab-ci.yml example

stages:
  - build
  - deploy
  - test
  - report

variables:
  CONTEXTQA_BASE_URL: "https://server.contextqa.com"
  CONTEXTQA_PLAN_ID: "<your_test_plan_id>"

build:
  stage: build
  script:
    - echo "Building application..."
    # Your build commands here

deploy_staging:
  stage: deploy
  script:
    - echo "Deploying to staging..."
    # Your deploy commands here

contextqa_tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    # Step 1: Authenticate and get Bearer token
    - |
      AUTH_RESPONSE=$(curl -s -X POST "${CONTEXTQA_BASE_URL}/api/v1/auth/login" \
        -H "Content-Type: application/json" \
        -d "{\"email\":\"${CONTEXTQA_USERNAME}\",\"password\":\"${CONTEXTQA_PASSWORD}\"}")
      TOKEN=$(echo "$AUTH_RESPONSE" | jq -r '.token')
      if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
        echo "ERROR: Failed to authenticate with ContextQA."
        exit 1
      fi
      echo "Authentication successful."

    # Step 2: Trigger the test plan (GET request)
    - |
      TRIGGER_RESPONSE=$(curl -s -X GET "${CONTEXTQA_BASE_URL}/api/v1/testplans/${CONTEXTQA_PLAN_ID}/execute" \
        -H "Authorization: Bearer ${TOKEN}")
      EXECUTION_ID=$(echo "$TRIGGER_RESPONSE" | jq -r '.executionId')
      if [ -z "$EXECUTION_ID" ] || [ "$EXECUTION_ID" = "null" ]; then
        echo "ERROR: Failed to trigger ContextQA test plan."
        echo "Response: $TRIGGER_RESPONSE"
        exit 1
      fi
      echo "ContextQA execution started: ${EXECUTION_ID}"
      echo "EXECUTION_ID=${EXECUTION_ID}" >> contextqa.env
      echo "TOKEN=${TOKEN}" >> contextqa.env

    # Step 3: Poll for completion
    - |
      source contextqa.env
      STATUS="RUNNING"
      ATTEMPT=0
      MAX_ATTEMPTS=60

      while [ "$STATUS" = "RUNNING" ] || [ "$STATUS" = "PENDING" ]; do
        if [ "$ATTEMPT" -ge "$MAX_ATTEMPTS" ]; then
          echo "ERROR: ContextQA execution timed out after $((MAX_ATTEMPTS * 30)) seconds."
          exit 1
        fi
        sleep 30
        ATTEMPT=$((ATTEMPT + 1))
        STATUS_RESPONSE=$(curl -s -X GET "${CONTEXTQA_BASE_URL}/api/v1/executions/${EXECUTION_ID}/status" \
          -H "Authorization: Bearer ${TOKEN}")
        STATUS=$(echo "$STATUS_RESPONSE" | jq -r '.status')
        echo "Attempt ${ATTEMPT}: ContextQA status = ${STATUS}"
      done

      REPORT_URL="https://app.contextqa.com/executions/${EXECUTION_ID}"
      echo "ContextQA final status: ${STATUS}"
      echo "Report: ${REPORT_URL}"

      echo "CONTEXTQA_STATUS=${STATUS}" >> contextqa.env
      echo "CONTEXTQA_REPORT_URL=${REPORT_URL}" >> contextqa.env

      if [ "$STATUS" = "FAILED" ]; then
        echo "ContextQA tests FAILED. See report: ${REPORT_URL}"
        exit 1
      fi

  artifacts:
    reports:
      dotenv: contextqa.env
    paths:
      - contextqa.env
    expire_in: 7 days
    when: always

publish_result:
  stage: report
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
  script:
    - echo "ContextQA Status: ${CONTEXTQA_STATUS}"
    - echo "Report URL: ${CONTEXTQA_REPORT_URL}"
    # Post result as MR comment if running in a merge request context
    - |
      if [ -n "$CI_MERGE_REQUEST_IID" ]; then
        curl -s -X POST \
          "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes" \
          -H "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" \
          -H "Content-Type: application/json" \
          -d "{\"body\":\"**ContextQA Test Result:** ${CONTEXTQA_STATUS}\\n\\nExecution report: ${CONTEXTQA_REPORT_URL}\"}"
        echo "MR comment posted."
      fi
  needs:
    - job: contextqa_tests
      artifacts: true
  when: always

How the pipeline works

Authentication: The before_script installs curl and jq on the Alpine base image. The first script block POSTs credentials to the ContextQA login endpoint. If authentication fails, the job exits immediately with code 1, which fails the pipeline stage.

Triggering the test plan: The execute endpoint uses a GET request. ContextQA returns the executionId in the response body. The execution ID and token are written to contextqa.env, a dotenv-format file that GitLab CI can pass between jobs using the artifacts: reports: dotenv mechanism.

Polling: The polling loop runs every 30 seconds with a 60-attempt ceiling (30-minute maximum). Adjust MAX_ATTEMPTS based on your suite's expected duration. The loop exits when the status transitions out of RUNNING or PENDING. A FAILED status exits with code 1, failing the pipeline stage and blocking any downstream jobs that depend on contextqa_tests.

Artifact passing: The contextqa.env dotenv file is published as a job artifact. The publish_result job uses needs: artifacts: true to receive the environment variables CONTEXTQA_STATUS and CONTEXTQA_REPORT_URL from contextqa_tests. This pattern works even when contextqa_tests fails, because when: always is set on publish_result.

MR comment: If the pipeline runs in a merge request context (CI_MERGE_REQUEST_IID is set), the publish_result job posts a comment to the MR with the test status and report URL. This requires a GITLAB_API_TOKEN CI/CD variable with API scope. If you do not want MR comments, remove the conditional block.

Storing the GitLab API token for MR comments

To post MR comments, add a fourth CI/CD variable:

1. Navigate to Settings → CI/CD → Variables.

2. Add a variable with Key GITLAB_API_TOKEN.

3. Set the value to a GitLab personal access token or project access token with api scope.

4. Mark it as Masked.

If you use a project access token, ensure it has the api role at minimum. Personal access tokens scoped to api also work but are tied to an individual user — prefer project access tokens for CI use.

Failing the pipeline on test failure

The job exits with code 1 when STATUS = "FAILED". In GitLab CI, a non-zero exit code from any script block fails the job and marks it as failed in the pipeline graph. Any downstream stage jobs that list contextqa_tests as a dependency will not run. This provides automatic release gating without additional configuration.

If you want the pipeline to continue after test failure (for example, to always run a cleanup stage), add allow_failure: true to the contextqa_tests job definition. The job will be marked as failed with a warning icon but will not block downstream jobs.

Frequently Asked Questions

Why does the script use GET to trigger the test plan instead of POST?

ContextQA's test plan execute endpoint is a GET request by API contract. The execution is fully identified by the plan ID in the URL path — there is no request body. Using POST will return a 405 Method Not Allowed. This is documented accurately in the ContextQA API reference.

How do I run ContextQA only on merge requests, not on every push?

Use GitLab's rules keyword on the contextqa_tests job:

contextqa_tests:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

This restricts the job to merge request pipelines only.

Can I run different test plans for different branches?

Yes. Use GitLab CI variables with branch-specific values, or use rules with variables overrides to set CONTEXTQA_PLAN_ID based on $CI_COMMIT_BRANCH.

Related

• GitHub Actions integration

• Jenkins integration

• CircleCI integration

• Azure DevOps integration

• Running tests and test plans

Connect ContextQA to your CI/CD pipeline in 15 minutes. Book a Demo → — See the full integration walkthrough for your existing toolchain.