Skip to main content

AI Failure Analysis

When a test fails, figuring out why can be the most time-consuming part of the QA process. ProvaLab.io AI automatically analyzes failures as they happen, giving you probable root causes, pattern detection, and actionable recommendations -- so you spend less time investigating and more time fixing.

How It Works​

When a test result is marked as Failed, ProvaLab.io AI examines:

  • Error messages and stack traces from the test execution
  • Screenshots and attachments captured during the run
  • Console and network logs (if available)
  • Historical data from previous runs of the same test
  • Related test results from the same test run or module

Based on this analysis, the AI provides a structured breakdown of what happened, why it likely happened, and what you can do about it.

Accessing Failure Analysis​

From the Test Results Page​

  1. Navigate to Test Runs and open a completed test run.
  2. Find a test result with a Failed status.
  3. Click on the failed test to view its details.
  4. Click the Analyze Failure button (marked with an AI icon).
  5. The AI analysis appears in a panel on the right side of the screen.

From the Dashboard​

When your dashboard shows recent failures, you can click on any failed test to go directly to its result details, then use the Analyze Failure button from there.

Automatic Analysis​

If your organization has enabled automatic failure analysis in Settings > AI Preferences, ProvaLab.io AI analyzes every failure as soon as it is recorded. You will see the analysis pre-populated when you open any failed test result.

What the Analysis Includes​

Root Cause Assessment​

The AI identifies the most likely reason the test failed and categorizes it:

  • Application Bug -- A genuine defect in the software being tested
  • Environment Issue -- Problems with the test environment (server down, service timeout, configuration error)
  • Test Data Issue -- Missing, stale, or incorrect test data
  • Test Script Issue -- A problem with the test itself (outdated selector, timing issue, incorrect assertion)
  • Infrastructure Issue -- Browser crash, device disconnection, network failure

Each root cause comes with a confidence level and an explanation of the reasoning.

Example Analysis:

Root Cause: Environment Issue (High Confidence)

The test "Verify checkout with credit card" failed at step 4 with PaymentGatewayTimeout: Connection refused on port 8443. This same error pattern appeared in 12 other tests during this run, all involving payment service calls. The payment gateway service in the staging environment is likely down or unreachable.

Similar Past Failures​

The AI searches your history for past instances of the same or similar failures:

  • Same test, same error -- How many times has this exact failure occurred before?
  • Different tests, same error -- Are other tests hitting the same underlying issue?
  • Resolution history -- If this failure was resolved before, what was the fix?

This helps you quickly determine whether you are dealing with a known issue or something new.

Flaky Test Detection​

ProvaLab.io AI monitors each test's pass/fail pattern over time and flags tests that behave inconsistently:

  • Flakiness score -- A percentage indicating how often the test produces inconsistent results
  • Flakiness pattern -- Does the test fail at certain times, on certain environments, or under certain conditions?
  • Recommendation -- Whether to quarantine, fix, or investigate the test

Example:

This test has a flakiness score of 35%. It has passed 13 times and failed 7 times in the last 20 runs. Failures correlate with runs executed between 2:00 AM and 4:00 AM UTC, suggesting a dependency on a scheduled maintenance window or batch process.

Risk Assessment​

For each failed test, the AI evaluates the broader risk to your application:

  • Affected features -- Which user-facing features are impacted by this failure
  • Severity estimate -- How critical is the affected functionality
  • Blast radius -- How many other tests or modules could be affected by the same root cause
  • Recommended priority -- Suggested priority for the fix based on impact analysis

Working with Failure Analysis​

Investigating a Single Failure​

  1. Open the failed test result.
  2. Click Analyze Failure.
  3. Read the root cause assessment.
  4. Check the similar past failures section to see if this is a known issue.
  5. Review the recommendation (fix the test, fix the app, check the environment, etc.).
  6. Take action: log a defect, update the test, or escalate to the relevant team.

Investigating Patterns Across a Test Run​

When multiple tests fail in the same run, the AI can identify shared root causes:

  1. Open the test run summary.
  2. Look for the Failure Patterns section (appears when there are 3 or more failures).
  3. The AI groups failures by common causes (e.g., "8 tests failed due to payment gateway timeout").
  4. Click on a pattern to see all affected tests and the recommended resolution.
tip

After a test run with many failures, check the Failure Patterns section first. If a single root cause explains most failures, fixing that one issue may resolve the majority of your failed tests.

Using Analysis to Improve Test Quality​

Over time, failure analysis data helps you improve your test suite:

  • Quarantine flaky tests -- Move consistently flaky tests to a separate suite so they do not block your pipeline.
  • Update brittle tests -- If AI frequently flags a test for "Test Script Issue," consider rewriting it with more robust selectors and waits.
  • Improve test data management -- If "Test Data Issue" is a common root cause, invest in better test data setup and teardown.
  • Strengthen environment monitoring -- If environment issues cause frequent failures, set up health checks for dependent services.

Flaky Test Management​

ProvaLab.io provides a dedicated view for managing flaky tests identified by AI:

  1. Navigate to Reports in the sidebar.
  2. Select Flaky Tests from the report options.
  3. You will see a list of all tests flagged as flaky, sorted by flakiness score.
  4. For each test, you can:
    • View history -- See the pass/fail pattern over time
    • Quarantine -- Remove the test from regular runs until it is fixed
    • Assign -- Assign the test to a team member for investigation
    • Suppress -- Mark the test as a known flaky test (failures will not trigger alerts)

Risk Dashboard​

The AI Risk Assessment is also available as a dashboard widget:

  1. Go to Reports and select Risk Assessment.
  2. View a heat map showing which modules and features carry the highest risk.
  3. Drill into any area to see the specific tests, failure rates, and AI recommendations.
tip

Review the Risk Assessment report before each release. It highlights the areas most likely to have issues, helping you prioritize your testing effort for maximum coverage of high-risk features.