CircleCI

CircleCI ContextQA integration — trigger test plans from a CircleCI config.yml, poll for results, and fail workflows automatically when tests fail.

Who is this for? SDETs, developers, and engineering managers who use CircleCI and want to run ContextQA test plans as part of their CI workflow and fail builds on test failures.

CircleCI ContextQA integration: A CI/CD configuration where a CircleCI workflow job triggers a ContextQA test plan via the REST API, polls for execution completion, and fails the CircleCI workflow when tests fail — providing automated release gating in CircleCI pipelines.

CircleCI workflows coordinate builds, deployments, and validation steps. Adding ContextQA as a dedicated job in your workflow ensures that every deployment to staging or production is validated by your full test suite before the workflow proceeds. This page provides a complete config.yml example, explains credential management in CircleCI, and covers result publication.

Prerequisites

Before configuring the CircleCI integration:

• A ContextQA account with at least one configured test plan

• The test plan ID (visible in the URL when viewing the plan: /test-plans/<plan_id>)

• A ContextQA service account (dedicated email and password — do not use a personal account)

• A CircleCI project connected to your repository

• CircleCI CLI or access to the CircleCI web interface for managing environment variables

Storing credentials as CircleCI environment variables

CircleCI environment variables are encrypted at rest and masked in job output. Store ContextQA credentials at the project level:

1. In CircleCI, navigate to your Project Settings.

2. Click Environment Variables in the left sidebar.

3. Click Add Environment Variable.

4. Set Name to CONTEXTQA_USERNAME and Value to your service account email. Click Add Environment Variable.

5. Repeat with Name CONTEXTQA_PASSWORD and your service account password.

For organization-wide reuse across multiple projects, store the variables in a CircleCI Context:

1. Navigate to Organization Settings → Contexts.

2. Create a context named contextqa (or any name that fits your naming convention).

3. Add CONTEXTQA_USERNAME and CONTEXTQA_PASSWORD to the context.

4. Reference the context in your workflow using the context key.

Complete config.yml example

version: 2.1

executors:
  contextqa-executor:
    docker:
      - image: cimg/base:stable
    resource_class: small

jobs:
  build:
    executor: contextqa-executor
    steps:
      - checkout
      - run:
          name: Build application
          command: echo "Build steps here"

  deploy_staging:
    executor: contextqa-executor
    steps:
      - run:
          name: Deploy to staging
          command: echo "Deploy steps here"

  contextqa_test:
    executor: contextqa-executor
    environment:
      CONTEXTQA_BASE_URL: https://server.contextqa.com
      CONTEXTQA_PLAN_ID: <your_test_plan_id>
    steps:
      - run:
          name: Authenticate with ContextQA
          command: |
            AUTH_RESPONSE=$(curl -s -X POST "${CONTEXTQA_BASE_URL}/api/v1/auth/login" \
              -H "Content-Type: application/json" \
              -d "{\"email\":\"${CONTEXTQA_USERNAME}\",\"password\":\"${CONTEXTQA_PASSWORD}\"}")
            TOKEN=$(echo "$AUTH_RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")
            if [ -z "$TOKEN" ]; then
              echo "ERROR: Authentication failed."
              exit 1
            fi
            echo "export CONTEXTQA_TOKEN=${TOKEN}" >> $BASH_ENV
            echo "Authentication successful."

      - run:
          name: Trigger ContextQA test plan
          command: |
            # Test plan execute endpoint uses GET, not POST
            TRIGGER_RESPONSE=$(curl -s -X GET "${CONTEXTQA_BASE_URL}/api/v1/testplans/${CONTEXTQA_PLAN_ID}/execute" \
              -H "Authorization: Bearer ${CONTEXTQA_TOKEN}")
            EXECUTION_ID=$(echo "$TRIGGER_RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['executionId'])")
            if [ -z "$EXECUTION_ID" ]; then
              echo "ERROR: Failed to start test plan execution."
              echo "Response: $TRIGGER_RESPONSE"
              exit 1
            fi
            echo "ContextQA execution started: ${EXECUTION_ID}"
            echo "export CONTEXTQA_EXECUTION_ID=${EXECUTION_ID}" >> $BASH_ENV

      - run:
          name: Poll for test completion
          command: |
            STATUS="RUNNING"
            ATTEMPT=0
            MAX_ATTEMPTS=60

            while [ "$STATUS" = "RUNNING" ] || [ "$STATUS" = "PENDING" ]; do
              if [ "$ATTEMPT" -ge "$MAX_ATTEMPTS" ]; then
                echo "ERROR: ContextQA timed out after $((MAX_ATTEMPTS * 30)) seconds."
                exit 1
              fi
              sleep 30
              ATTEMPT=$((ATTEMPT + 1))
              STATUS_RESPONSE=$(curl -s -X GET \
                "${CONTEXTQA_BASE_URL}/api/v1/executions/${CONTEXTQA_EXECUTION_ID}/status" \
                -H "Authorization: Bearer ${CONTEXTQA_TOKEN}")
              STATUS=$(echo "$STATUS_RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['status'])")
              echo "Attempt ${ATTEMPT}: ContextQA status = ${STATUS}"
            done

            REPORT_URL="https://app.contextqa.com/executions/${CONTEXTQA_EXECUTION_ID}"
            echo "Final status: ${STATUS}"
            echo "Report: ${REPORT_URL}"

            # Write result to a file for the artifact step
            mkdir -p /tmp/contextqa-results
            echo "Status: ${STATUS}" > /tmp/contextqa-results/result.txt
            echo "Execution ID: ${CONTEXTQA_EXECUTION_ID}" >> /tmp/contextqa-results/result.txt
            echo "Report URL: ${REPORT_URL}" >> /tmp/contextqa-results/result.txt

            echo "export CONTEXTQA_STATUS=${STATUS}" >> $BASH_ENV
            echo "export CONTEXTQA_REPORT_URL=${REPORT_URL}" >> $BASH_ENV

            if [ "$STATUS" = "FAILED" ]; then
              echo "ContextQA tests FAILED. See report: ${REPORT_URL}"
              exit 1
            fi

      - store_artifacts:
          path: /tmp/contextqa-results
          destination: contextqa-results

workflows:
  build_test_deploy:
    jobs:
      - build
      - deploy_staging:
          requires:
            - build
      - contextqa_test:
          requires:
            - deploy_staging
          context:
            - contextqa

How the config.yml works

Executor: The cimg/base:stable image includes curl and Python 3, which are used for HTTP calls and JSON parsing respectively. The small resource class is sufficient — the CircleCI job itself does nothing computationally intensive; it only issues HTTP requests and sleeps between polls.

Environment variables: The CONTEXTQA_BASE_URL and CONTEXTQA_PLAN_ID are set as non-secret environment variables directly in the job definition. The secret credentials (CONTEXTQA_USERNAME, CONTEXTQA_PASSWORD) come from the contextqa CircleCI Context referenced in the workflow definition.

$BASH_ENV for variable passing: CircleCI steps within the same job share environment through $BASH_ENV. Writing export VAR=value >> $BASH_ENV makes VAR available in all subsequent steps in the same job. The CONTEXTQA_TOKEN, CONTEXTQA_EXECUTION_ID, and final status variables are all passed this way.

GET request for test plan execution: The trigger step uses curl -X GET. This is correct — ContextQA's execute endpoint is a GET request. Using POST to this endpoint returns a 405 error.

Polling logic: The polling step sleeps 30 seconds between attempts and allows a maximum of 60 attempts (30 minutes total). Adjust MAX_ATTEMPTS to suit your suite's typical run time. The loop exits when status is neither RUNNING nor PENDING. A FAILED status causes the step to exit with code 1, which fails the CircleCI job.

Workflow dependency: The contextqa_test job lists deploy_staging in its requires block. CircleCI will not start the test job until the deploy job completes successfully. If the deploy fails, the test job is skipped automatically.

Storing the execution result as a CircleCI artifact

The store_artifacts step publishes the /tmp/contextqa-results/result.txt file to CircleCI. This file contains the execution status, execution ID, and report URL. After the job completes, navigate to the Artifacts tab of the job in the CircleCI dashboard to find the file.

The artifact is stored regardless of whether the job passes or fails because store_artifacts always runs — CircleCI artifact steps are not gated by prior step success. This ensures that even when tests fail, the report URL is accessible from the CircleCI job artifacts.

Using a CircleCI Context for shared credentials

If multiple projects in your CircleCI organization need to access ContextQA, use a Context rather than per-project environment variables:

1. Navigate to Organization Settings → Contexts → Create Context.

2. Name the context contextqa.

3. Add CONTEXTQA_USERNAME and CONTEXTQA_PASSWORD.

4. In each project's config.yml, reference the context under the workflow job definition using the context key as shown in the example.

Contexts are governed by security groups in CircleCI. You can restrict which projects and team members can use the contextqa context, preventing unauthorized access to your test infrastructure.

Failing the workflow on test failure

When the contextqa_test job fails (exit code 1 from the polling step), CircleCI marks the job as failed and the overall workflow status becomes failed. Any jobs in the workflow that list contextqa_test in their requires block will not run. This provides automatic release gating: a production deploy job that requires the test job will never run if tests fail.

Frequently Asked Questions

Why use Python 3 for JSON parsing instead of jq?

The cimg/base:stable image ships Python 3 by default. jq is not always present and would require an installation step. Python's standard library json module is reliable and avoids adding dependencies. If you prefer jq, add sudo apt-get install -y jq to a run step before the authentication step.

Can I reuse the ContextQA token across multiple pipeline runs?

No. Obtain a fresh token at the start of each job. ContextQA tokens have a limited lifetime, and storing a token in a CircleCI environment variable that persists across builds would eventually result in authentication failures.

How do I trigger a ContextQA test plan on a schedule in CircleCI rather than on every push?

Use CircleCI's scheduled pipeline feature. In the CircleCI web UI, navigate to Project Settings → Triggers and create a scheduled trigger. Set the trigger to call your pipeline on the desired schedule. The contextqa_test job will run as part of that scheduled workflow without requiring a code push.

Can I run multiple test plans in the same CircleCI workflow?

Yes. Add additional jobs following the contextqa_test pattern, each with a different CONTEXTQA_PLAN_ID. List them as parallel jobs under the workflow, or chain them sequentially using requires if you need them to run in order.

Related

• GitHub Actions integration

• Jenkins integration

• GitLab CI integration

• Azure DevOps integration

• Running tests and test plans

Connect ContextQA to your CI/CD pipeline in 15 minutes. Book a Demo → — See the full integration walkthrough for your existing toolchain.