CI/CD Pipeline Integration

Run your TMS-integrated tests in CI/CD pipelines. This guide covers GitHub Actions, Jenkins, and GitLab CI with complete, copy-paste-ready configurations.

Overview

Running TMS-integrated tests in CI is straightforward:

Set TMS environment variables as secrets in your CI platform
Install dependencies (including the TMS SDK)
Run tests exactly as you would locally

TMS reporters detect the environment variables and report results automatically. No code changes needed beyond what you already have for local development.

Environment Variables

All CI platforms need these secrets configured:

Variable	Description
`TMS_API_URL`	Your TMS instance URL
`TMS_API_KEY`	API key from TMS Settings
`TMS_ORGANIZATION_ID`	Your organization ID
`TMS_PROJECT_ID`	Your project ID
`TMS_ENVIRONMENT`	Set to `ci` or the branch name (Node, Java, .NET and Ruby SDKs; the pytest plugin has no environment setting, use `--tms-test-run-name`)

Naming your CI test runs

Set TMS_TEST_RUN_NAME to include useful CI context:

TMS_TEST_RUN_NAME="CI: ${GITHUB_REF_NAME} #${GITHUB_RUN_NUMBER}"

This makes it easy to find CI runs in the TMS dashboard.

GitHub Actions

Playwright + TMS

.github/workflows/e2e-tests.yml
name: E2E Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  TMS_API_URL: ${{ secrets.TMS_API_URL }}
  TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
  TMS_ORGANIZATION_ID: ${{ secrets.TMS_ORGANIZATION_ID }}
  TMS_PROJECT_ID: ${{ secrets.TMS_PROJECT_ID }}
  TMS_ENVIRONMENT: ci
  TMS_TEST_RUN_NAME: "CI: ${{ github.ref_name }} #${{ github.run_number }}"

jobs:
  playwright:
    runs-on: ubuntu-latest
    timeout-minutes: 30

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        run: npx playwright test

      - name: Upload Playwright report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

Jest + TMS

.github/workflows/unit-tests.yml
name: Unit Tests

on:
  push:
    branches: [main, develop]
  pull_request:

env:
  TMS_API_URL: ${{ secrets.TMS_API_URL }}
  TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
  TMS_ORGANIZATION_ID: ${{ secrets.TMS_ORGANIZATION_ID }}
  TMS_PROJECT_ID: ${{ secrets.TMS_PROJECT_ID }}
  TMS_ENVIRONMENT: ci

jobs:
  jest:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: ['18', '20']

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run Jest tests
        run: npx jest --ci --coverage

      - name: Upload coverage
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage-node-${{ matrix.node-version }}
          path: coverage/

Cypress + TMS

.github/workflows/cypress-tests.yml
name: Cypress Tests

on:
  push:
    branches: [main]
  pull_request:

env:
  TMS_API_URL: ${{ secrets.TMS_API_URL }}
  TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
  TMS_ORGANIZATION_ID: ${{ secrets.TMS_ORGANIZATION_ID }}
  TMS_PROJECT_ID: ${{ secrets.TMS_PROJECT_ID }}
  TMS_ENVIRONMENT: ci

jobs:
  cypress:
    runs-on: ubuntu-latest
    timeout-minutes: 30

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run Cypress tests
        uses: cypress-io/github-action@v6
        with:
          browser: chrome
          headed: false

      - name: Upload screenshots on failure
        uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: cypress-screenshots
          path: cypress/screenshots

      - name: Upload videos
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: cypress-videos
          path: cypress/videos

pytest + TMS

.github/workflows/python-tests.yml
name: Python Tests

on:
  push:
    branches: [main, develop]
  pull_request:

env:
  TMS_API_URL: ${{ secrets.TMS_API_URL }}
  TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
  TMS_ORGANIZATION_ID: ${{ secrets.TMS_ORGANIZATION_ID }}
  TMS_PROJECT_ID: ${{ secrets.TMS_PROJECT_ID }}
  # The pytest plugin reads TMS_BASE_URL (defaults to https://tms.provalab.io)
  TMS_BASE_URL: ${{ secrets.TMS_API_URL }}

jobs:
  pytest:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install https://plugins.provalab.io/packages/pytest_tms_plugin-1.0.13-py3-none-any.whl

      - name: Run pytest with TMS
        # Reporting is on by default once TMS_* is set (there is no --tms-enable).
        # The pytest plugin has no environment flag; tag the run name instead.
        run: pytest tests/ -v --tms-test-run-name="ci #${{ github.run_number }}"

Java (Maven) + TMS

.github/workflows/java-tests.yml
name: Java Tests

on:
  push:
    branches: [main]
  pull_request:

env:
  TMS_API_URL: ${{ secrets.TMS_API_URL }}
  TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
  TMS_ORGANIZATION_ID: ${{ secrets.TMS_ORGANIZATION_ID }}
  TMS_PROJECT_ID: ${{ secrets.TMS_PROJECT_ID }}
  TMS_ENVIRONMENT: ci

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'
          cache: 'maven'

      - name: Run tests
        run: mvn test -B

Setting up GitHub Secrets

Go to your repository on GitHub
Navigate to Settings > Secrets and variables > Actions
Click New repository secret for each:

Secret Name	Value
`TMS_API_URL`	`https://tms.yourcompany.com`
`TMS_API_KEY`	Your API key
`TMS_ORGANIZATION_ID`	Your organization ID (number)
`TMS_PROJECT_ID`	Your project ID (number)

Organization-level secrets

If multiple repositories need TMS access, set secrets at the GitHub Organization level (Organization Settings > Secrets) so all repos inherit them.

Jenkins

Declarative Pipeline

Jenkinsfile
pipeline {
    agent any

    environment {
        TMS_API_URL     = credentials('tms-api-url')
        TMS_API_KEY     = credentials('tms-api-key')
        TMS_ORGANIZATION_ID = credentials('tms-org-id')
        TMS_PROJECT_ID  = credentials('tms-project-id')
        TMS_ENVIRONMENT = 'ci'
        TMS_TEST_RUN_NAME = "Jenkins: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
    }

    stages {
        stage('Install') {
            steps {
                sh 'npm ci'
            }
        }

        stage('Unit Tests') {
            steps {
                sh 'npx jest --ci'
            }
        }

        stage('E2E Tests') {
            steps {
                sh 'npx playwright install --with-deps'
                sh 'npx playwright test'
            }
            post {
                always {
                    archiveArtifacts artifacts: 'playwright-report/**', allowEmptyArchive: true
                }
            }
        }
    }

    post {
        always {
            junit '**/test-results/*.xml'
        }
    }
}

Parallel E2E Tests

Jenkinsfile
pipeline {
    agent any

    environment {
        TMS_API_URL     = credentials('tms-api-url')
        TMS_API_KEY     = credentials('tms-api-key')
        TMS_ORGANIZATION_ID = credentials('tms-org-id')
        TMS_PROJECT_ID  = credentials('tms-project-id')
        TMS_ENVIRONMENT = 'ci'
    }

    stages {
        stage('Install') {
            steps {
                sh 'npm ci'
                sh 'npx playwright install --with-deps'
            }
        }

        stage('E2E Tests') {
            parallel {
                stage('Playwright - Chrome') {
                    environment {
                        TMS_TEST_RUN_NAME = "Jenkins Playwright Chrome #${env.BUILD_NUMBER}"
                    }
                    steps {
                        sh 'npx playwright test --project=chromium'
                    }
                }
                stage('Cypress - Chrome') {
                    environment {
                        TMS_TEST_RUN_NAME = "Jenkins Cypress Chrome #${env.BUILD_NUMBER}"
                    }
                    steps {
                        sh 'npx cypress run --browser chrome'
                    }
                }
            }
        }
    }
}

Setting up Jenkins Credentials

Go to Jenkins > Manage Jenkins > Manage Credentials
Add credentials (type: Secret text) for each:
- tms-api-url
- tms-api-key
- tms-org-id
- tms-project-id

GitLab CI

Basic Configuration

.gitlab-ci.yml
variables:
  TMS_API_URL: $TMS_API_URL
  TMS_API_KEY: $TMS_API_KEY
  TMS_ORGANIZATION_ID: $TMS_ORGANIZATION_ID
  TMS_PROJECT_ID: $TMS_PROJECT_ID
  TMS_ENVIRONMENT: ci

stages:
  - test

unit-tests:
  stage: test
  image: node:20
  script:
    - npm ci
    - npx jest --ci
  artifacts:
    when: always
    reports:
      junit: test-results/junit.xml

e2e-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.42.0-jammy
  script:
    - npm ci
    - npx playwright test
  artifacts:
    when: always
    paths:
      - playwright-report/
    expire_in: 30 days

python-tests:
  stage: test
  image: python:3.11
  script:
    - pip install -r requirements.txt
    - pip install https://plugins.provalab.io/packages/pytest_tms_plugin-1.0.13-py3-none-any.whl
    # Reporting is on by default once TMS_* is set (no --tms-enable); name the run instead of an environment flag.
    - pytest tests/ -v --tms-test-run-name="GitLab CI #$CI_PIPELINE_IID"

Parallel E2E Tests

.gitlab-ci.yml
e2e-playwright:
  stage: test
  image: mcr.microsoft.com/playwright:v1.42.0-jammy
  variables:
    TMS_TEST_RUN_NAME: "GitLab Playwright #$CI_PIPELINE_IID"
  parallel:
    matrix:
      - PROJECT: [chromium, firefox, webkit]
  script:
    - npm ci
    - npx playwright test --project=$PROJECT
  artifacts:
    when: always
    paths:
      - playwright-report/

e2e-cypress:
  stage: test
  image: cypress/browsers:latest
  variables:
    TMS_TEST_RUN_NAME: "GitLab Cypress #$CI_PIPELINE_IID"
  script:
    - npm ci
    - npx cypress run --browser chrome
  artifacts:
    when: on_failure
    paths:
      - cypress/screenshots/
      - cypress/videos/

Setting up GitLab CI/CD Variables

Go to your GitLab project
Navigate to Settings > CI/CD > Variables
Click Add variable for each:

Key	Value	Options
`TMS_API_URL`	`https://tms.yourcompany.com`	Protected
`TMS_API_KEY`	Your API key	Protected, Masked
`TMS_ORGANIZATION_ID`	Your org ID	Protected
`TMS_PROJECT_ID`	Your project ID	Protected

Mark sensitive values as "Masked"

Always mask TMS_API_KEY so it doesn't appear in CI logs.

Best Practices

Use descriptive test run names

Include CI context in the run name so you can trace runs back to specific builds:

# GitHub Actions
TMS_TEST_RUN_NAME="CI: ${GITHUB_REF_NAME} #${GITHUB_RUN_NUMBER}"

# Jenkins
TMS_TEST_RUN_NAME="Jenkins: ${JOB_NAME} #${BUILD_NUMBER}"

# GitLab CI
TMS_TEST_RUN_NAME="GitLab: ${CI_COMMIT_REF_NAME} #${CI_PIPELINE_IID}"

Tag the environment

Always set TMS_ENVIRONMENT to distinguish CI runs from local development:

TMS_ENVIRONMENT=ci

The pytest plugin does not read TMS_ENVIRONMENT; put the environment in the run name with --tms-test-run-name="ci #$BUILD_NUMBER" instead.

Don't let TMS failures break your build

TMS reporter SDKs are designed to fail gracefully. If TMS is unreachable, tests still run normally. However, you can explicitly disable TMS if needed:

TMS_DISABLED=true   # Skip all TMS reporting

Separate test runs for different suites

If you run multiple test suites in a single CI pipeline, each suite automatically creates its own TMS test run. Use TMS_TEST_RUN_NAME to differentiate them:

- name: Run unit tests
  env:
    TMS_TEST_RUN_NAME: "Unit Tests #${{ github.run_number }}"
  run: npx jest

- name: Run E2E tests
  env:
    TMS_TEST_RUN_NAME: "E2E Tests #${{ github.run_number }}"
  run: npx playwright test

Verify It Works

Configure secrets in your CI platform
Add or update your CI configuration file
Push a commit to trigger the pipeline
Check the CI logs for the TMS banner (Test Run ID)
Open TMS and verify the test run appears with environment tag "ci"

Common issues

"Not configured" in CI logs: Secrets are not being passed to the job. Check that variable names match exactly
Results appearing as "local" not "ci": Make sure TMS_ENVIRONMENT=ci is set
Multiple runs created: This is expected if you run multiple test commands. Each creates its own TMS test run
API key exposed in logs: Mark TMS_API_KEY as a secret/masked variable in your CI platform

Overview​

Environment Variables​

GitHub Actions​

Playwright + TMS​

Jest + TMS​

Cypress + TMS​

pytest + TMS​

Java (Maven) + TMS​

Setting up GitHub Secrets​

Jenkins​

Declarative Pipeline​

Parallel E2E Tests​

Setting up Jenkins Credentials​

GitLab CI​

Basic Configuration​

Parallel E2E Tests​

Setting up GitLab CI/CD Variables​

Best Practices​

Use descriptive test run names​

Tag the environment​

Don't let TMS failures break your build​

Separate test runs for different suites​

Verify It Works​

Overview

Environment Variables

GitHub Actions

Playwright + TMS

Jest + TMS

Cypress + TMS

pytest + TMS

Java (Maven) + TMS

Setting up GitHub Secrets

Jenkins

Declarative Pipeline

Parallel E2E Tests

Setting up Jenkins Credentials

GitLab CI

Basic Configuration

Parallel E2E Tests

Setting up GitLab CI/CD Variables

Best Practices

Use descriptive test run names

Tag the environment

Don't let TMS failures break your build

Separate test runs for different suites

Verify It Works