Skip to main content

AI from the CLI

ProvaLab.io has a built-in AI assistant that can generate test cases, analyze failures, detect flaky tests, generate documentation, and answer natural language questions about your test data. All of these features are available directly from the command line.

Chat with the AI assistant​

Send a message to the AI agent and get an instant response:

tms ai chat --message "What were the most common failures this week?"

Example output:

AI Assistant:

╭───────────────────────────────────────────────────────────────────╮
│ Based on test execution data from the last 7 days, here are the  │
│ most common failure patterns:                                     │
│                                                                   │
│ 1. **Timeout errors** (34 occurrences) -- primarily in the       │
│    checkout flow API tests                                        │
│ 2. **Assertion failures** (21 occurrences) -- response body      │
│    mismatches in the user profile endpoints                       │
│ 3. **Connection refused** (12 occurrences) -- intermittent       │
│    failures pointing to staging environment instability           │
│                                                                   │
│ The timeout errors correlate with a deployment on Wednesday       │
│ that increased average response times by 40%.                     │
╰───────────────────────────────────────────────────────────────────╯

Suggestions:
  - Review the checkout flow API for performance regressions
  - Update user profile test assertions to match new response format
  - Check staging environment health and resource allocation

You can provide additional context to narrow the scope:

tms ai chat --message "Show me test coverage for the authentication module" \
  --context "project_id=1"

Generate test cases​

Generate AI-powered test cases for an API endpoint:

tms ai generate test-cases \
  --endpoint-id 5 \
  --count 10 \
  --test-type all

The --test-type flag controls what kind of test cases are generated:

ValueDescription
positiveHappy-path tests with valid inputs
negativeTests with invalid or missing inputs
edgeBoundary values and edge cases
securityAuthentication, authorization, and injection tests
allA mix of all types (default)

Example output:

 SUCCESS  Generated 10 test cases

Test Case 1:
  Name: Valid login with correct credentials
  Type: positive
  Expected Status: 200
  Description: Verify user can authenticate with valid email and password

Test Case 2:
  Name: Login with empty password
  Type: negative
  Expected Status: 422
  Description: Verify proper validation error when password is empty

Test Case 3:
  Name: Login with SQL injection in email field
  Type: security
  Expected Status: 422
  Description: Verify the API rejects SQL injection attempts in the email parameter

Generate a load test script​

Have the AI write a K6 load test script for you:

tms ai generate load-script \
  --scenario-id 1 \
  --test-type stress

The available test types are load, stress, spike, and soak. The AI generates a complete K6 script that you can review, edit, and optionally save directly to the scenario:

 SUCCESS  K6 script generated successfully

Generated Script:

   1 │ import http from 'k6/http';
   2 │ import { check, sleep } from 'k6';
   3 │
   4 │ export const options = {
   5 │   stages: [
   6 │     { duration: '2m', target: 100 },
   7 │     { duration: '5m', target: 500 },
   8 │     { duration: '2m', target: 1000 },
   9 │     { duration: '5m', target: 1000 },
  10 │     { duration: '2m', target: 0 },
  11 │   ],
  12 │ };
  ...

Save script to scenario? [y/n]:

Analyze test failures​

Investigate what went wrong in a test run with AI-powered root cause analysis:

tms ai analyze failures --run-id 42 --run-type api

Example output:

 Analyzing test failures...

AI Analysis:

╭───────────────────────────────────────────────────────────────────╮
│ ## Root Cause Analysis                                           │
│                                                                   │
│ The 3 failures in run #42 share a common pattern: all are        │
│ timeout errors on endpoints that query the `orders` table.       │
│ This correlates with a missing database index on                 │
│ `orders.customer_id`, which was dropped in migration #0034.      │
│                                                                   │
│ **Confidence:** 87%                                               │
╰───────────────────────────────────────────────────────────────────╯

Identified Root Causes:
  - Missing index on orders.customer_id causing full table scans
  - Query timeout threshold (5s) too aggressive for current data volume

Recommendations:
  - Re-create the index: CREATE INDEX idx_orders_customer_id ON orders(customer_id)
  - Increase query timeout to 10s as a temporary workaround
  - Add a performance test case for order lookup latency

For load test failures, use --run-type load:

tms ai analyze failures --run-id 100 --run-type load

Detect flaky tests​

Identify tests that pass and fail intermittently:

tms ai analyze flaky-tests --project-id 1 --days 30

Example output:

 Found 3 flaky tests:

 - User profile update
  Flakiness Score: 0.85
  Pass Rate: 70.0%
  Total Runs: 50
  Reason: Intermittent timeout on upstream user service

 - Shopping cart total calculation
  Flakiness Score: 0.62
  Pass Rate: 82.0%
  Total Runs: 40
  Reason: Race condition between cart update and price recalculation

 - Email notification delivery
  Flakiness Score: 0.55
  Pass Rate: 85.0%
  Total Runs: 60
  Reason: Dependency on external SMTP service with variable latency

The flakiness score ranges from 0 (stable) to 1 (extremely flaky). Tests with a score above 0.5 deserve investigation.

Generate API documentation​

Generate Markdown documentation from an API collection:

tms ai document --collection-id 1

Save it to a file:

tms ai document --collection-id 1 --output api-docs.md

Natural language queries​

Ask questions about your test data in plain English. The AI translates your question into SQL, runs it safely (read-only), and shows the results:

tms ai query --question "How many tests failed in the last week?"

Example output:

Generated SQL:

SELECT COUNT(*) as failed_count
FROM test_results
WHERE status = 'failed'
  AND executed_at >= NOW() - INTERVAL '7 days';

Results:

┌──────────────┐
│ failed_count │
├──────────────┤
│           47 │
└──────────────┘

Add project context for more targeted results:

tms ai query \
  --question "What is the pass rate trend for the last 4 sprints?" \
  --project-id 1

AI-powered test plan features​

The CLI also provides AI features specific to test plans:

Smart test selection​

Let the AI suggest which tests to run based on code changes:

tms test-plan ai-suggest 1 \
  --organization-id 1 \
  --changed-files "src/auth/login.py,src/auth/oauth.py" \
  --max-tests 20

Coverage gap analysis​

Identify areas that are not adequately covered:

tms test-plan ai-gaps 1 --organization-id 1

Risk assessment​

Get an AI-generated risk score and mitigation recommendations:

tms test-plan ai-risks 1 --organization-id 1

Next steps​