Tutorial: CI/CD with GitHub Actions

Set up automated test execution in your GitHub Actions pipeline — organize tests into a suite and plan, configure GitHub secrets, add the workflow file, and verify your first automated run.

Who is this for? Developers, SDETs, and engineering managers who have test cases in ContextQA and want to run them automatically on every pull request or push — so the system catches broken builds before they reach production.

Running tests manually after each deployment works when your team ships once a week. When you ship multiple times a day, manual testing becomes a bottleneck. CI/CD integration solves this by triggering your ContextQA test plan every time code changes — automatically, in the background, with results reported directly to the pull request.

In this tutorial you:

1. Organize test cases into a suite designed for CI
2. Create a test plan that the pipeline triggers
3. Store ContextQA credentials as GitHub secrets
4. Add a GitHub Actions workflow file to your repository
5. Push a change and verify the automated run

End result: Every pull request to your main branch triggers a ContextQA test plan. The workflow blocks the merge if tests fail, and posts a link to the full execution report.

Prerequisites

• A ContextQA account with at least two test cases created (quickstart if you need to create your first)
• A GitHub repository where you can add workflow files
• Admin or write access to the repository's Settings → Secrets page
• Familiarity with creating test cases and running tests

---

Step 1: Create a CI smoke test suite

CI pipelines should run fast. Instead of running every test in your workspace, create a focused suite that covers your application's critical paths — login, core feature, and one end-to-end flow.

1. In the left sidebar, click the Test Development icon.
2. Click the Test Suites tab.
3. Click + Create Test Suite.
4. Enter a suite name: Smoke_CI.
5. Add a description: Critical-path tests for CI/CD pipeline execution. Keep this suite under 10 test cases for fast feedback.
6. Click Create Suite.

The system creates an empty suite. Now add your most important test cases:

1. Open the Smoke_CI suite.
2. Click + Add Test Cases.
3. Select two to five test cases that cover your application's critical paths. Good candidates:

   • A login or authentication flow
   • The primary user action (for example, search, create a record, submit a form)
   • A navigation flow that touches multiple pages
4. Click Add to confirm.

Tip: Keep CI suites small and fast. A 5-test suite that runs in 2 minutes gives developers feedback while their PR is still fresh. Save large regression suites for nightly scheduled runs.

Verify it worked: The Smoke_CI suite shows your selected test cases in the list. The test case count badge matches the number you added.

---

Step 2: Create a test plan for CI execution

A test plan wraps one or more suites with execution configuration — browser, environment, and parallel settings. The CI pipeline triggers this plan by ID.

1. Navigate to Test Development → Test Plans.
2. Click + Create Test Plan.
3. Fill in the plan configuration:

Field	Value	Why
Plan name	CI — Smoke Tests	Prefix with "CI" so the team knows this plan is pipeline-triggered
Test suites	Select Smoke_CI	The suite you created in step 1
Browser	Chrome	Chrome is the most common CI target; add more browsers later
Environment	Select your staging environment	CI tests should run against staging, not production

1. Click Save.

After saving, note the test plan ID. You can find it in two places:

• In the browser URL bar when viewing the plan: https://app.contextqa.com/...test-plans/<plan_id>
• In the plan details panel

Copy this ID — you need it for the GitHub Actions workflow.

Verify it worked: Open the test plan and click Run to execute it manually once. Confirm all test cases pass against your staging environment before wiring it into CI. If tests fail here, fix them first — a failing test plan in CI blocks every pull request.

---

Step 3: Store credentials as GitHub secrets

The GitHub Actions workflow needs your ContextQA credentials to authenticate with the API. Store them as encrypted secrets so they never appear in code or logs.

1. In your GitHub repository, navigate to Settings → Secrets and variables → Actions.
2. Click New repository secret and add:

Secret name	Value
CONTEXTQA_USERNAME	Your ContextQA account email
CONTEXTQA_PASSWORD	Your ContextQA account password

1. Click New repository variable (under the Variables tab) and add:

Variable name	Value
CONTEXTQA_PLAN_ID	The test plan ID from step 2

Important: Use a dedicated ContextQA service account for CI rather than your personal login. This lets you revoke CI access independently and keeps the audit trail clean. Create a service account under Administration → Team Management with the minimum required permissions.

Verify it worked: After saving, the secrets page shows CONTEXTQA_USERNAME and CONTEXTQA_PASSWORD in the secrets list (values are hidden) and CONTEXTQA_PLAN_ID in the variables list.

---

Step 4: Add the GitHub Actions workflow file

Create a workflow file in your repository that triggers the ContextQA test plan on every pull request to main.

1. In your repository, create the directory .github/workflows/ if it does not exist.
2. Create a new file: .github/workflows/contextqa-tests.yml
3. Paste the following workflow:

name: Run ContextQA Tests

on:
  pull_request:
    branches: [main]

jobs:
  test:
    name: ContextQA Smoke Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run ContextQA test plan
        env:
          CONTEXTQA_USERNAME: ${{ secrets.CONTEXTQA_USERNAME }}
          CONTEXTQA_PASSWORD: ${{ secrets.CONTEXTQA_PASSWORD }}
        run: |
          pip install requests
          python -c "
          import requests, os, time, sys

          username = os.environ['CONTEXTQA_USERNAME']
          password = os.environ['CONTEXTQA_PASSWORD']
          plan_id = '${{ vars.CONTEXTQA_PLAN_ID }}'

          # Authenticate
          auth = requests.post(
              'https://server.contextqa.com/api/v1/auth/login',
              json={'username': username, 'password': password},
              timeout=30
          )
          auth.raise_for_status()
          token = auth.json()['token']
          headers = {'Authorization': f'Bearer {token}'}

          # Trigger the test plan
          run = requests.get(
              f'https://server.contextqa.com/api/v1/testplans/{plan_id}/execute',
              headers=headers,
              timeout=30
          )
          run.raise_for_status()
          execution_id = run.json()['executionId']
          print(f'Execution started: {execution_id}')

          # Poll for results (up to 15 minutes)
          for attempt in range(30):
              time.sleep(30)
              status = requests.get(
                  f'https://server.contextqa.com/api/v1/executions/{execution_id}/status',
                  headers=headers,
                  timeout=30
              )
              status.raise_for_status()
              result = status.json().get('status', 'RUNNING')
              print(f'Attempt {attempt + 1}: {result}')

              if result == 'PASSED':
                  print(f'All tests passed.')
                  print(f'Report: https://app.contextqa.com/executions/{execution_id}')
                  sys.exit(0)
              elif result in ['FAILED', 'ERROR', 'ABORTED']:
                  print(f'Tests failed: {result}')
                  print(f'Report: https://app.contextqa.com/executions/{execution_id}')
                  sys.exit(1)

          print('Timeout: execution did not complete within 15 minutes.')
          sys.exit(1)
          "

1. Commit and push this file to a branch (not main yet — you test it in the next step).

How the workflow works

The workflow follows three stages:

1. Authenticate — sends your credentials to the ContextQA login endpoint and receives a Bearer token
2. Trigger — calls the test plan execute endpoint (a GET request) and receives an execution ID
3. Poll — checks the execution status every 30 seconds until it completes or times out

If the test plan passes, the workflow exits with code 0 (success). If it fails, the workflow exits with code 1, which marks the GitHub Actions check as failed.

Common mistake: The ContextQA execute endpoint uses a GET request, not POST. Using POST returns a 405 Method Not Allowed error. This is by design — the plan ID in the URL fully identifies the execution.

---

Step 5: Push a change and verify the automated run

Now test the full pipeline by opening a pull request.

1. Create a new branch from main:

git checkout -b test/verify-ci-pipeline

1. Make a small change to any file (for example, add a comment to your README).
2. Commit and push the branch:

git add .
git commit -m "test: verify ContextQA CI pipeline"
git push -u origin test/verify-ci-pipeline

1. Open a pull request targeting main on GitHub.
2. Navigate to the Checks tab on the pull request. You should see the ContextQA Smoke Tests job running.

Watch the job output. It logs each polling attempt with the current status:

Execution started: 12345
Attempt 1: RUNNING
Attempt 2: RUNNING
Attempt 3: PASSED
All tests passed.
Report: https://app.contextqa.com/executions/12345

1. Click the report link in the job output to open the full ContextQA execution results — screenshots, video, and per-step pass/fail status for every test case in the plan.

Verify it worked: The pull request shows a green checkmark next to the ContextQA Smoke Tests check. The job output includes the execution report URL.

---

Step 6: Block merges on test failure (optional)

To require ContextQA tests to pass before a pull request can be merged:

1. In your GitHub repository, go to Settings → Branches → Branch protection rules.
2. Click Add branch protection rule (or edit an existing rule for main).
3. Enable Require status checks to pass before merging.
4. Search for ContextQA Smoke Tests and select it.
5. Click Save changes.

With this rule active, the Merge button on pull requests stays disabled until the ContextQA workflow job passes. Developers see a clear message: "Required status check — ContextQA Smoke Tests" with a pass or fail indicator.

---

Summary

You set up automated CI/CD testing in six steps:

1. Created a CI smoke suite with your most critical test cases
2. Created a test plan targeting that suite with browser and environment configuration
3. Stored credentials as GitHub encrypted secrets
4. Added a workflow file that authenticates, triggers the plan, and polls for results
5. Verified the pipeline by opening a pull request and watching the automated run
6. Blocked merges on test failure using branch protection rules

Every pull request to main now runs your ContextQA tests automatically. Developers get fast feedback, and broken code cannot merge without explicit override.

Next steps

• Add more triggers: Extend the workflow to also trigger on pushes to main or develop by adding push: branches: [main, develop] under the on: section. See the GitHub Actions reference for the full example.
• Run across multiple browsers: Add a matrix strategy to run the same plan on Chrome, Firefox, and Safari in parallel. See the matrix builds example.
• Post results as a PR comment: Add a step that posts the execution report URL directly to the pull request as a comment. See the PR comment example.
• Set up nightly regression runs: Create a larger test plan with your full regression suite and trigger it on a cron schedule. See Scheduling.
• Try other CI platforms: ContextQA integrates with Jenkins, GitLab CI, and CircleCI using the same API pattern.

Related pages

• Quickstart Guide — create your first test case
• Managing Test Suites — suite organization and naming conventions
• Running Tests — all execution options for test cases, suites, and plans
• Environments — configure base URLs and variables for different targets
• GitHub Actions — full GitHub Actions reference with advanced patterns
• Jenkins — Jenkins pipeline integration
• Integrations Overview — all supported CI/CD and tool integrations

Automate your test pipeline in 15 minutes — no code required. Start Free Trial → — Or Book a Demo → to see CI/CD integration with your existing toolchain.