Skip to main content

TestNG Integration

Run your TestNG tests as usual and have results automatically reported to TMS. The SDK provides a comprehensive listener with full lifecycle management, log capture, and QA Infrastructure integration.

Install​

Maven​

The SDK is served from the ProvaLab Maven repository, not Maven Central, so add the repository as well as the dependency:

pom.xml
<repositories>
    <repository>
        <id>provalab</id>
        <url>https://plugins.provalab.io/maven</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>com.tms</groupId>
        <artifactId>tms-java-sdk</artifactId>
        <version>2.2.0</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Gradle​

build.gradle
repositories {
    maven { url 'https://plugins.provalab.io/maven' }
}
dependencies {
    testImplementation 'com.tms:tms-java-sdk:2.2.0'
}

Requirements: Java 17+, TestNG 7.8+

Configure​

1. Set environment variables​

export TMS_API_KEY=your-api-key
export TMS_API_URL=https://tms.yourcompany.com
export TMS_ORGANIZATION_ID=1
export TMS_PROJECT_ID=12

2. Register the TMS listener​

Option A: In testng.xml (recommended)

src/test/resources/testng.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="My Test Suite">
    <listeners>
        <listener class-name="com.tms.testng.TMSTestNGListener"/>
    </listeners>

    <test name="Login Tests">
        <classes>
            <class name="com.example.LoginTest"/>
        </classes>
    </test>
</suite>

Option B: Via annotation

import com.tms.testng.TMSTestNGListener;
import org.testng.annotations.Listeners;

@Listeners(TMSTestNGListener.class)
public class LoginTest {
    // ...
}

Option C: Programmatically

import com.tms.testng.TMSTestNGListener;
import org.testng.TestNG;

TestNG testng = new TestNG();
testng.addListener(new TMSTestNGListener());
testng.setTestClasses(new Class[]{LoginTest.class});
testng.run();

3. Annotate your tests with TMS case IDs​

src/test/java/com/example/LoginTest.java
import com.tms.testng.TMSTestCase;
import org.testng.annotations.Test;
import static org.testng.Assert.*;

public class LoginTest {

    @Test
    @TMSTestCase(id = 101)
    public void shouldLoginWithValidCredentials() {
        boolean result = AuthService.login("[email protected]", "password123");
        assertTrue(result, "Login should succeed");
    }

    @Test
    @TMSTestCase(id = 102, priority = "high", testType = "regression")
    public void shouldRejectInvalidPassword() {
        assertThrows(AuthException.class, () -> {
            AuthService.login("[email protected]", "wrong-password");
        });
    }

    @Test
    @TMSTestCase(
        id = 103,
        title = "SSO Login Flow",
        priority = "critical",
        testType = "e2e",
        tags = {"auth", "sso"},
        description = "Verify SSO redirect and token exchange"
    )
    public void shouldLoginWithSSO() {
        SSOResult result = AuthService.loginWithSSO("okta");
        assertNotNull(result.getToken());
    }
}

Run​

Maven​

mvn test

With testng.xml​

mvn test -DsuiteXmlFile=src/test/resources/testng.xml

Gradle​

gradle test

You'll see TMS output in the console:

========================================
  TMS TestNG Integration
========================================
  Test Run ID: 856
  Suite: My Test Suite
  Project ID: 12
  Organization ID: 1
========================================

  [Login Tests]
    [PASS] shouldLoginWithValidCredentials (0.12s)
    [PASS] shouldRejectInvalidPassword (0.08s)
    [FAIL] shouldLoginWithSSO (2.31s)

========================================
  TMS Test Run Summary
========================================
  Test Run ID: 856
  Total:   3
  Passed:  2
  Failed:  1
  Skipped: 0
  Status:  FAILED
========================================

View Results in TMS​

  1. Open TMS and go to your project
  2. Click Test Runs in the sidebar
  3. Find the run named "TestNG: My Test Suite - 2026-02-20 14:30:00"
  4. Click into it to see individual test results with error details, execution logs, and metadata

The @TMSTestCase Annotation​

@TMSTestCase(
    id = 456,                              // Required: TMS test case ID
    title = "User Login with SSO",         // Optional: display name
    priority = "critical",                 // Optional: critical, high, medium, low
    testType = "smoke",                    // Optional: smoke, regression, e2e, unit
    tags = {"auth", "sso"},                // Optional: categorization tags
    description = "Verify SSO login flow", // Optional: test description
    componentId = 5,                       // Optional: TMS component ID
    expectedDurationSeconds = 30,          // Optional: expected runtime
    automated = true                       // Optional: is this automated? (default: true)
)

Advanced Configuration​

DataProvider support​

The listener handles @DataProvider iterations, creating separate TMS results for each:

@DataProvider(name = "loginData")
public Object[][] loginData() {
    return new Object[][] {
        {"[email protected]", "admin123", true},
        {"[email protected]", "user123", true},
        {"[email protected]", "wrong", false},
    };
}

@Test(dataProvider = "loginData")
@TMSTestCase(id = 201)
public void shouldHandleLogin(String email, String password, boolean expected) {
    assertEquals(AuthService.login(email, password), expected);
}

Each iteration appears as a separate result in TMS, with the parameters shown in the test name.

Test case mapping file​

Instead of annotations, you can use a JSON mapping file:

test-case-mapping.json
{
  "com.example.LoginTest": {
    "shouldLoginWithValidCredentials": { "id": 101, "priority": "high" },
    "shouldRejectInvalidPassword": { "id": 102, "priority": "medium" }
  }
}
export TMS_MAPPING_FILE=test-case-mapping.json

WebDriver integration​

If tests use Selenium WebDriver, the listener automatically captures screenshots, network logs, and browser type:

import com.tms.testng.TMSTestContextManager;

public class UITest {
    private WebDriver driver;

    @BeforeMethod
    public void setup() {
        driver = new ChromeDriver();
        // Register the driver with TMS for auto-capture
        TMSTestContextManager.getCurrentContext().setWebDriver(driver);
    }

    @AfterMethod
    public void teardown() {
        if (driver != null) driver.quit();
    }
}

Log capture​

The listener installs a java.util.logging handler that forwards application logs to TMS. These appear in the "Execution Logs" tab for each test result.

Disable TMS for local development​

export TMS_DISABLED=true
mvn test

Custom test run name​

export TMS_TEST_RUN_NAME="Nightly Regression - TestNG"

Verify It Works​

  1. Add the dependency and configure testng.xml
  2. Set your environment variables
  3. Create a simple test:
src/test/java/com/example/TMSVerifyTest.java
import com.tms.testng.TMSTestCase;
import org.testng.annotations.Test;
import static org.testng.Assert.*;

public class TMSVerifyTest {

    @Test
    @TMSTestCase(id = 1)
    public void verifyTmsIntegration() {
        assertEquals(1 + 1, 2, "TMS integration working");
    }
}
  1. Run: mvn test -Dtest=TMSVerifyTest
  2. Check the console for the TMS summary
  3. Open TMS and confirm the test run appears
Common issues
  • "Not configured" message: Make sure TMS_API_KEY, TMS_ORGANIZATION_ID, and TMS_PROJECT_ID environment variables are set
  • Listener not running: Verify the listener is registered in testng.xml or via @Listeners
  • Health check warning: The "TMS health check failed" warning is non-fatal -- results are still reported