Skip to main content

Robot Framework Integration

Report Robot Framework test results to TMS using a listener. The TMS listener hooks into Robot's execution lifecycle and sends results as tests complete.

Not yet published: do not pip install tms-plugin

The tms-plugin Python package this page describes is not published on PyPI or on plugins.provalab.io yet. The name is unclaimed on PyPI, so installing it by name would fetch whatever a third party uploads under it. This page documents the listener so you can prepare your suites; it cannot be installed yet. Check the plugin catalogue for the official install path when it ships.

Install​

Not yet available. When published, the package will be installable from plugins.provalab.io; the listener below is documented so you can prepare your suites.

Requirements (once published): Python >= 3.9, Robot Framework >= 5.0

Configure​

1. Tag your test cases with TMS IDs​

Use the tms:<id> tag format to link Robot test cases to TMS:

tests/login.robot
*** Settings ***
Library    SeleniumLibrary

*** Test Cases ***
Login With Valid Credentials
    [Tags]    tms:101    smoke    critical
    Open Browser    https://app.yourcompany.com/login    chrome
    Input Text    id=email    [email protected]
    Input Text    id=password    password123
    Click Button    id=login-btn
    Page Should Contain    Dashboard
    [Teardown]    Close Browser

Login With Invalid Password
    [Tags]    tms:102    regression
    Open Browser    https://app.yourcompany.com/login    chrome
    Input Text    id=email    [email protected]
    Input Text    id=password    wrong-password
    Click Button    id=login-btn
    Page Should Contain    Invalid credentials
    [Teardown]    Close Browser

Login With Expired Account
    [Tags]    tms:103    regression
    Open Browser    https://app.yourcompany.com/login    chrome
    Input Text    id=email    [email protected]
    Input Text    id=password    password123
    Click Button    id=login-btn
    Page Should Contain    Account expired
    [Teardown]    Close Browser
Tag format

You can use either tms:123 or tms_case_id:123 as the tag format. Both are recognized by the listener.

2. Run with the TMS listener​

Option A: Pass configuration via Robot variables

robot --listener tms_plugin.integrations.robot_listener.TMSListener \
      --variable TMS_URL:https://tms.yourcompany.com \
      --variable TMS_API_KEY:your-api-key \
      --variable TMS_ORGANIZATION_ID:1 \
      --variable TMS_TEST_RUN_ID:123 \
      tests/

Option B: Use environment variables

export TMS_URL=https://tms.yourcompany.com
export TMS_API_KEY=your-api-key
export TMS_ORGANIZATION_ID=1
export TMS_TEST_RUN_ID=123

robot --listener tms_plugin.integrations.robot_listener.TMSListener tests/

Option C: Auto-create test run from test plan

robot --listener tms_plugin.integrations.robot_listener.TMSListener \
      --variable TMS_URL:https://tms.yourcompany.com \
      --variable TMS_API_KEY:your-api-key \
      --variable TMS_ORGANIZATION_ID:1 \
      --variable TMS_TEST_PLAN_ID:10 \
      tests/

Run​

robot --listener tms_plugin.integrations.robot_listener.TMSListener tests/

The listener reports results as each test finishes. After the suite completes, TMS metadata is added to Robot's output:

==============================================================================
Login Tests
==============================================================================
Login With Valid Credentials                                          | PASS |
Login With Invalid Password                                           | PASS |
Login With Expired Account                                            | FAIL |
Account expired message not found
==============================================================================
Login Tests                                                           | 2 passed, 1 failed
==============================================================================

TMS results appear in Robot's output.xml metadata:

TMS Test Run ID: 854
TMS Total Tests: 3
TMS Passed: 2
TMS Failed: 1
TMS Pass Rate: 66.67%

View Results in TMS​

  1. Open TMS and go to your project
  2. Click Test Runs in the sidebar
  3. Find the run (by ID or name "Robot Run - Login Tests - 2026-02-20 14:30:00")
  4. Click into it to see individual test results with error messages

Advanced Configuration​

Constructor arguments​

You can pass configuration directly to the listener constructor:

robot --listener tms_plugin.integrations.robot_listener.TMSListener:base_url=https://tms.yourcompany.com:api_key=your-key:organization_id=1:test_run_id=123 tests/

Environment selection​

robot --listener tms_plugin.integrations.robot_listener.TMSListener \
      --variable TMS_ENVIRONMENT:staging \
      tests/

Screenshot capture​

The listener looks for screenshots in the screenshots/ directory that match the test name. If you're using SeleniumLibrary, configure it to save screenshots there:

*** Settings ***
Library    SeleniumLibrary    screenshot_root_directory=screenshots

Configuration precedence​

The listener resolves configuration in this order (first wins):

  1. Constructor arguments
  2. Robot variables (--variable TMS_API_KEY:...)
  3. Environment variables (TMS_API_KEY=...)

Verify It Works​

  1. Install the plugin (not yet published, see the notice at the top of this page)

  2. Create a simple test:

tests/tms_verify.robot
*** Test Cases ***
Verify TMS Integration
    [Tags]    tms:1
    Log    TMS integration is working
    Should Be True    ${True}
  1. Run it:
robot --listener tms_plugin.integrations.robot_listener.TMSListener \
      --variable TMS_URL:https://tms.yourcompany.com \
      --variable TMS_API_KEY:your-api-key \
      --variable TMS_ORGANIZATION_ID:1 \
      --variable TMS_TEST_RUN_ID:1 \
      tests/tms_verify.robot
  1. Check the output for TMS metadata
  2. Open TMS and confirm the test result appears under your test run
Common issues
  • "TMS API key not provided": Make sure TMS_API_KEY is set as a Robot variable or environment variable
  • Tests run but nothing appears in TMS: Make sure tests have the tms:<id> tag
  • "Either test_run_id or test_plan_id must be provided": Pass one of TMS_TEST_RUN_ID or TMS_TEST_PLAN_ID