pytest Integration

Run your pytest tests as usual and have results, logs, screenshots and (with QA Infrastructure) video reported to TMS automatically. The plugin registers itself through pytest's plugin entry point, so there is nothing to switch on: once it is installed and configured, every run reports.

Install

The plugin is distributed only from plugins.provalab.io as a wheel. It is not on PyPI, so install it by URL:

pip install --upgrade https://plugins.provalab.io/packages/pytest_tms_plugin-1.0.13-py3-none-any.whl

Requirements: Python >= 3.8, pytest >= 7.0.0

Do not pip install tms-plugin or pip install pytest-tms-plugin

Neither name is published on PyPI by ProvaLab. Installing by name resolves against the public index and would pull whatever a third party uploads under that name. Always install from the plugins.provalab.io URL above, and check the plugin catalogue for the latest version.

To pin it in requirements.txt:

requirements.txt

pytest-tms-plugin @ https://plugins.provalab.io/packages/pytest_tms_plugin-1.0.13-py3-none-any.whl

Configure

1. Set up credentials

The plugin reads environment variables first; command-line flags override them. Reporting is on by default whenever the three required values are present. There is no --tms-enable flag; --tms-disabled (or TMS_DISABLED=true) turns reporting off.

Option A: Environment variables (recommended for CI)

export TMS_API_KEY=tms_...            # create one at https://auth.provalab.io/api-keys
export TMS_ORGANIZATION_ID=1          # numeric TMS organization id
export TMS_PROJECT_ID=12              # numeric project id
export TMS_TEST_PLAN_ID=10            # optional: attach runs to a test plan

The ProvaLab cloud URL https://tms.provalab.io is the default. Self-hosted instances set TMS_BASE_URL (the pytest plugin reads TMS_BASE_URL, not the TMS_API_URL used by the other SDKs).

Option B: Command-line flags

pytest --tms-api-key=tms_... \
       --tms-org-id=1 \
       --tms-project-id=12 \
       --tms-test-plan-id=10

Option C: Non-secret settings in pytest.ini, the key from the environment

pytest.ini

[pytest]
addopts = --tms-org-id=1 --tms-project-id=12 --tms-test-plan-id=10

TMS_API_KEY=tms_... pytest

Always set --tms-test-plan-id for release runs

The plugin creates a new test run every time pytest starts. With --tms-test-plan-id (or TMS_TEST_PLAN_ID) that run is attached to the plan, so the plan's coverage counters update. Without it the plan stays at tests_executed = 0.

Finding your ids

Both ids are the integers TMS uses in its own URLs: open the project in TMS (the browser URL is /app/projects/<PROJECT_ID>/...) and the API calls the UI makes go to /api/v1/organizations/<ORG_ID>/.... Do not use the organization UUID shown on auth.provalab.io: --tms-org-id is an integer and the plugin rejects a UUID.

2. Link your tests to TMS test cases

tests/test_login.py

import pytest
from pytest_tms import tms_test_case

@tms_test_case(id=101)
def test_login_with_valid_credentials():
    """Verify user can log in with correct email and password."""
    result = login("[email protected]", "password123")
    assert result.success is True

@pytest.mark.tms_test_case(id=102)   # the plain pytest marker works too
def test_login_with_invalid_password():
    """Verify login fails with wrong password."""
    with pytest.raises(AuthenticationError):
        login("[email protected]", "wrong-password")

Alternatively, map pytest node ids to case ids in a mapping file (default test_case_mapping.json in the project root; override with TMS_MAPPING_FILE):

test_case_mapping.json

{
  "tests/test_login.py::test_login_with_valid_credentials": 101,
  "tests/test_login.py::test_login_with_invalid_password": 102
}

Unlinked tests create new cases

Tests without a case id are still reported: TMS creates a new case for each one on first run, which piles up duplicates across releases. Link every test before you ship a suite to CI.

Run

pytest

The terminal summary shows the run link and counts:

========================= TMS Integration =========================
  Test Run #852: https://tms.provalab.io/app/projects/12/test-runs/852
  Results: 2 passed, 1 failed, 0 skipped (3 total)
  Test Results: 3 reported to TMS

To run without reporting (for example locally), pass --tms-disabled or set TMS_DISABLED=true.

View Results in TMS

1. Open TMS and go to your project
2. Click Test Runs in the sidebar
3. The newest run is yours (use --tms-test-run-name to give it a recognisable name)
4. Click into it to see individual results with error tracebacks, logs and screenshots

Advanced Configuration

Browsers from QA Infrastructure (Selenium / Appium)

Use the plugin's driver fixture instead of creating your own WebDriver. It provisions a browser (or mobile device) on QA Infrastructure, streams live view, records video, captures a screenshot when the test fails and attaches everything to the test result:

tests/test_ui.py

from pytest_tms import tms_test_case

@tms_test_case(id=201)
def test_dashboard_loads(driver):
    driver.get("https://app.example.com/dashboard")
    assert "Dashboard" in driver.title
    # If this fails, the failure screenshot is uploaded to the result automatically

Options: --tms-infra-platform (linux, android, ios), --tms-infra-browser (chrome, firefox, chrome_mobile, safari_mobile), --tms-device-type (emulator or physical), --headless, and --tms-app-path / --tms-app-package / --tms-app-activity for native mobile apps. The QA Infrastructure URL defaults to https://infra.provalab.io; override with --qa-infra-url / QA_INFRA_URL and --qa-infra-api-key / QA_INFRA_API_KEY (falls back to TMS_API_KEY).

Playwright mode

pytest --tms-playwright --tms-browser=chromium --tms-trace=retain-on-failure

Use the playwright_page fixture in your tests to get a page on a QA Infrastructure browser. Video (--tms-record-video, on by default; --tms-no-video to disable), traces (--tms-trace: off, on, retain-on-failure), viewport (--tms-viewport=1280x720) and live view (--tms-no-live-view, --tms-live-view-fps) are set per run, or per test with @pytest.mark.playwright_config(...). --tms-local-browser runs a local Playwright browser instead of a QA Infrastructure one.

Naming runs per environment

The plugin has no environment flag. Put the environment (and the CI build number) in the run name instead:

pytest --tms-test-run-name="staging #${BUILD_NUMBER}"

Security gates

--tms-security-scan NAME (repeatable, or TMS_SECURITY_SCANS=a,b) runs a ProvaLab Security Portal scan at session end and reports a security_gate[NAME] result; --tms-security-gate makes pytest exit 1 when a gate fails. The scan categories are described on the MCP tools page.

All command-line options

Every flag below comes from pytest --help with pytest-tms-plugin 1.0.13 installed. Flags that take a value override the environment variable shown.

TMS reporting

Flag	Env var	Default	Description
--tms-api-key	TMS_API_KEY	-	API key (required)
--tms-org-id	TMS_ORGANIZATION_ID	-	Numeric organization id (required)
--tms-project-id	TMS_PROJECT_ID	-	Numeric project id (required)
--tms-test-plan-id	TMS_TEST_PLAN_ID	-	Attach the run to this test plan
--tms-test-run-name	-	auto-generated	Custom name for the run
--tms-component-id	TMS_COMPONENT_ID	-	Component id (optional)
--tms-url	TMS_BASE_URL	https://tms.provalab.io	TMS base URL (self-hosted only)
--tms-frontend-url	TMS_FRONTEND_URL	same as --tms-url	URL used for run links in the summary
--tms-disabled	TMS_DISABLED=true	off	Turn reporting off
--tms-version	TMS_VERSION	-	Version name or id to link security runs to
--tms-security-scan NAME	TMS_SECURITY_SCANS	-	Security Portal scan to run at session end (repeatable)
--tms-security-gate	-	off	Exit 1 when any security gate fails
--tms-security-url	TMS_SECURITY_URL	-	Security Portal base URL
--tms-security-timeout	TMS_SECURITY_TIMEOUT	30m	Max wait per scan

Environment-only settings: TMS_MAPPING_FILE, TMS_VERIFY_SSL (default true), TMS_LOG_BATCH_SIZE, TMS_LOG_BATCH_INTERVAL.

QA Infrastructure (Selenium / Appium)

Flag	Default	Description
--qa-infra-url	QA_INFRA_URL, else https://infra.provalab.io	QA Infrastructure API URL
--qa-infra-api-key	QA_INFRA_API_KEY, else TMS_API_KEY	QA Infrastructure API key
--headless	off	Run the browser headless
--enable-video	on	Record video of the session
--tms-infra-platform	linux	linux, android or ios
--tms-infra-browser	chrome	chrome, chrome_mobile, safari_mobile or firefox
--tms-device-type	emulator	emulator or physical (USB-tethered device)
--tms-app-path	-	APK/IPA to install for native app tests
--tms-app-package	-	Android package name
--tms-app-activity	-	Android main activity

Playwright

Flag	Default	Description
--tms-playwright	off	Enable Playwright mode (instead of Selenium)
--tms-browser	chromium	chromium, firefox or webkit
--tms-headless / --tms-headed	headless	Headless or headed browser
--tms-record-video / --tms-no-video	video on	Record video
--tms-trace	retain-on-failure	off, on or retain-on-failure
--tms-viewport	1280x720	Viewport as WIDTHxHEIGHT
--tms-slow-mo	0	Slow down operations by N milliseconds
--tms-local-browser	off	Use a local Playwright browser (skip QA Infrastructure)
--tms-gateway-url	-	Playwright gateway WebSocket URL
--tms-live-view / --tms-no-live-view	on	Real-time screenshot streaming
--tms-live-view-fps	5	Live view frames per second

Verify It Works

1. Install the plugin:

pip install --upgrade https://plugins.provalab.io/packages/pytest_tms_plugin-1.0.13-py3-none-any.whl

2. Create a simple test:

tests/test_tms_verify.py

from pytest_tms import tms_test_case

@tms_test_case(id=1)
def test_verify_tms_integration():
    """Verify TMS integration is working."""
    assert 1 + 1 == 2

3. Run it:

pytest tests/test_tms_verify.py \
    --tms-api-key=tms_... \
    --tms-org-id=1 \
    --tms-project-id=12

4. Check the terminal for the TMS Integration section with the run link
5. Open the link and confirm the result appears in the run

Common issues

• pytest: error: unrecognized arguments: --tms-enable: the plugin has no --tms-enable; reporting is on by default and --tms-disabled turns it off. Use only the flags in the tables above.
• A warning that TMS is not configured: set TMS_API_KEY, TMS_ORGANIZATION_ID and TMS_PROJECT_ID (or the matching flags)
• --tms-org-id: invalid int value or "looks like a UUID": use the numeric TMS organization id, not the auth.provalab.io organization UUID
• Plan still shows 0 tests executed: add --tms-test-plan-id
• Tests run but nothing appears in TMS: check the API key is valid and, for self-hosted, that TMS_BASE_URL is reachable