Skip to main content

Running Tests

The CLI supports three types of test execution: test plans (manual/automated test cases), API testing (HTTP endpoint collections), and load testing (performance scenarios). This page covers all three.

Test plans​

Test plans group test cases for a particular release, sprint, or testing cycle.

List test plans​

tms test-plan list --organization-id 1

You can filter by project, status, type, or priority:

tms test-plan list --organization-id 1 \
  --project-id 2 \
  --status active \
  --type regression \
  --priority high

Example output:

                                 Test Plans
┌────┬────────────────────────┬────────────┬────────┬──────────┬───────────┬──────────┐
│ ID │ Name                   │ Type       │ Status │ Priority │ Pass Rate │ Progress │
├────┼────────────────────────┼────────────┼────────┼──────────┼───────────┼──────────┤
│  1 │ Sprint 10 Regression   │ regression │ active │ high     │ 85.0%     │ 60%      │
│  2 │ Smoke Tests v2.1       │ smoke      │ draft  │ critical │ --        │ 0%       │
│  3 │ UAT - Payment Module   │ uat        │ active │ high     │ 92.5%     │ 80%      │
└────┴────────────────────────┴────────────┴────────┴──────────┴───────────┴──────────┘

Total: 3 test plans

Execute a test plan​

tms test-plan execute 1 --organization-id 1

You can specify the environment and a custom run name:

tms test-plan execute 1 \
  --organization-id 1 \
  --environment staging \
  --name "Pre-release Regression"

Example output:

 SUCCESS  Test plan execution started
  Test Run ID: 42
  Environment: staging
  Status: running

View test plan metrics​

tms test-plan metrics 1 --organization-id 1

Add --detailed for risk assessment and timeline information:

tms test-plan metrics 1 --organization-id 1 --detailed

Add test cases to a plan​

tms test-plan add-tests 1 --organization-id 1 --test-cases 10,11,12,13

API testing​

API testing lets you run HTTP endpoint collections and validate responses, status codes, and performance.

List API collections​

tms api collection list --project-id 1

Example output:

                     API Collections (Project 1)
┌────┬────────────────────┬──────────────────────────┬───────────┬────────────┐
│ ID │ Name               │ Base URL                 │ Auth Type │ Created    │
├────┼────────────────────┼──────────────────────────┼───────────┼────────────┤
│  1 │ User API           │ https://api.example.com  │ bearer    │ 2025-12-01 │
│  2 │ Payment Gateway    │ https://pay.example.com  │ api_key   │ 2025-12-15 │
└────┴────────────────────┴──────────────────────────┴───────────┴────────────┘

Create a collection​

tms api collection create \
  --project-id 1 \
  --name "User API" \
  --base-url "https://api.example.com" \
  --auth-type bearer

Import from Postman or OpenAPI​

# Import a Postman collection
tms api collection import \
  --project-id 1 \
  --file my_collection.postman.json \
  --format postman

# Import an OpenAPI spec
tms api collection import \
  --project-id 1 \
  --file openapi.json \
  --format openapi

Example output:

 SUCCESS  Collection imported successfully (ID: 3)
  Endpoints discovered: 24
  Test cases generated: 48

Run API tests​

tms api run start --collection-id 1 --environment staging

Add --wait to block until the run completes and display results inline:

tms api run start --collection-id 1 --environment prod --wait

Example output with --wait:

 Running tests... done

 STATUS     completed
 PASSED     22
 FAILED     2
 TOTAL      24
 DURATION   8.4s

View test results​

tms api run results 42

Add --detailed for a per-test breakdown:

tms api run results 42 --detailed
                         Test Results
┌──────────────┬────────┬──────────────────┬──────────────────────┐
│ Test Case ID │ Status │ Response Time    │ Error                │
├──────────────┼────────┼──────────────────┼──────────────────────┤
│           1  │ passed │ 120ms            │                      │
│           2  │ passed │ 95ms             │                      │
│           3  │ failed │ 3200ms           │ Timeout exceeded     │
│           4  │ passed │ 145ms            │                      │
└──────────────┴────────┴──────────────────┴──────────────────────┘

Security scanning​

Run OWASP-style security scans against your API endpoints:

tms api security scan --endpoint-id 5 --scan-type owasp --wait

View the report afterward:

tms api security report 12

Load testing​

Load testing validates how your application performs under traffic using K6 under the hood.

List scenarios​

tms load scenario list --project-id 1

Create a scenario​

tms load scenario create \
  --project-id 1 \
  --name "Checkout Flow Load Test" \
  --target-url "https://shop.example.com/checkout" \
  --test-type load \
  --vus 500 \
  --duration 300 \
  --ramp-up 30

Upload a custom K6 script​

tms load scenario upload-script \
  --scenario-id 1 \
  --script checkout_test.js

Run a load test​

tms load run start --scenario-id 1

Add --wait for a live metrics dashboard in your terminal:

tms load run start --scenario-id 1 --wait

While the test runs you will see a live-updating table:

                Load Test Metrics
┌─────────────────────┬──────────────────┐
│ Metric              │ Value            │
├─────────────────────┼──────────────────┤
│ Current VUs         │ 485              │
│ RPS                 │ 1,247.50         │
│ Avg Response Time   │ 142ms            │
│ P95 Response Time   │ 380ms            │
│ Error Rate          │ 0.12%            │
│ Total Requests      │ 62,375           │
└─────────────────────┴──────────────────┘

View results after completion​

tms load run results 100

Example output:

Load Test Results:
  Status: completed
  Peak Users: 500
  Duration: 300s
  Total Requests: 187,500
  Failed Requests: 225
  Avg Response Time: 142ms
  P95 Response Time: 380ms
  P99 Response Time: 720ms
  RPS: 625.0
  Error Rate: 0.12%

Add --detailed to include detected bottlenecks:

tms load run results 100 --detailed

View time-series metrics​

tms load run metrics 100 --limit 50

Analyze bottlenecks with AI​

tms load analyze bottlenecks 100

Compare runs side by side​

tms load analyze compare --run-ids 100,95
                     Load Test Comparison
┌────────┬───────────────────┬─────────────────┬─────────┬────────────┐
│ Run ID │ Avg Response Time │ P95 Response    │ RPS     │ Error Rate │
├────────┼───────────────────┼─────────────────┼─────────┼────────────┤
│    95  │ 165ms             │ 420ms           │ 580.0   │ 0.18%      │
│   100  │ 142ms             │ 380ms           │ 625.0   │ 0.12%     │
└────────┴───────────────────┴─────────────────┴─────────┴────────────┘

Exporting results​

Export any results to a file for reporting or CI/CD artifact storage:

# Export load test results as JSON
tms load run results 100 --output results.json

# Export API results as CSV
tms api run results 42 --format csv > api_results.csv

Using the CLI in CI/CD​

The CLI is designed for automation. Here is a GitHub Actions example:

- name: Run API Tests
  run: |
    tms api run start \
      --collection-id ${{ secrets.API_COLLECTION_ID }} \
      --environment ci \
      --wait
  env:
    TMS_API_KEY: ${{ secrets.TMS_API_KEY }}
    TMS_API_URL: ${{ secrets.TMS_API_URL }}
tip

When running in CI/CD, always set TMS_API_KEY and TMS_API_URL as environment variables rather than using tms auth login. This avoids storing credentials on disk.

Next steps​

  • AI from the CLI -- Generate tests, analyze failures, and chat with the AI agent
  • API Reference -- Full REST API documentation for building custom integrations