Slide 1: One failed test. Five possible causes.

A practical guide to automation failure analysis by Siddhartha Mukherjee, Principal Quality Engineer.

Slide 2: A failure is the start of an investigation

A missing invoice after 30 seconds is an observation. Investigate whether processing failed, remains pending, or used the wrong record.

Slide 3: Five possible causes

Consider product behaviour, testware, data, setup and environment. Classifications may overlap and must follow evidence.

Slide 4: Evidence that makes a failure reproducible

Preserve execution context, unique record identity, failure timestamps, expected behaviour and actual results.

Slide 5: A fictional import failure

In this fictional asynchronous import, the job remains RUNNING at 30 seconds and reports SUCCEEDED at 42 seconds. The evidence supports an early validation hypothesis. Check the documented processing contract before concluding.

Slide 6: AI assistance and human verification

AI can assemble timelines and suggest hypotheses. People verify source accuracy, record identity, requirements, business impact and fix results.

Slide 7: Correction and validation

Poll the exact job with a bounded deadline, handle terminal rejection, respect idempotency, and verify the intended business result.

Slide 8: A reusable investigation checklist

Preserve the first result, identify the earliest deviation, test hypotheses, choose a focused correction, and verify affected regression paths.

Further reading

Read the companion article: One Failed Test, Five Possible Causes.

Author: Siddhartha Mukherjee. This presentation expresses general quality engineering practices and does not imply employer endorsement.

The fastest tests are not automatically the most valuable. The newest feature is not the only risk. Running everything labelled “smoke” may leave a changed dependency untested.

Risk-based regression is the practice of choosing and ordering checks according to the failures that matter most for this release. It makes the reasoning behind reduced coverage explicit.

This article uses a fictional release and illustrative time estimates. It is a planning example, not an account of an employer's delivery process.

1. Start with the change, then follow its reach

Imagine a customer-management application with three changes:

• A payment-calculation update.
• A change to a shared role-permission helper.
• A new optional field in a bulk import.

Testing only those screens would miss their connections. Payment calculations affect invoices and balances. A shared permission helper may affect unchanged pages. An import change may affect existing layouts or child-record mapping.

Build a small impact map from the changed components to business journeys, roles, integrations, and data. Ask developers and feature owners to challenge it. Label uncertain dependencies as questions to resolve, rather than silently treating them as unaffected.

2. Describe risks as failures with consequences

“Test payments” is a scope label. “An incorrect rounding change produces the wrong amount due” is a risk statement that helps select checks.

For each candidate risk, record:

Use qualitative priorities or a simple scoring scheme consistently. A numerical score can structure discussion, but it does not make subjective estimates precise. Do not let a low total score hide a non-negotiable security, regulatory, or data-integrity requirement.

3. Select checks that challenge each risk

A fictional selection might look like this:

These are examples, not a universal minimum suite. Actual cases must follow the application's requirements and supported configurations.

As discussed in A Test Passed—But Did It Actually Verify Anything?, every selected check needs assertions that support its claim. Coverage by name alone is weak evidence.

4. Budget for usable evidence, not just execution

Two hours of execution followed by no time to investigate results is not a complete review window.

An illustrative plan for our release is:

Validate estimates against actual run history, setup costs, and available people. Parallel execution helps only when resources and data are independent; otherwise, it can introduce collisions and reduce the credibility of results.

If essential checks cannot fit, escalate the constraint. Options include delaying the decision, reducing release scope, or obtaining additional capacity. Compressing the plan does not reduce the underlying risk.

5. Order checks to reveal important problems early

First establish that the candidate build, environment, and prerequisites are suitable. Then run the highest-impact checks with a realistic chance of exposing release-blocking problems.

A failure in a critical shared permission path may change the value of continuing broad UI regression. An unavailable prerequisite may invalidate dependent tests until restored.

This does not mean stopping all independent testing after the first failure. Separate blocked journeys from checks that can still provide valid evidence, and communicate any replanning.

Retain a small, justified set of checks for important unchanged behaviour and unexpected interactions. Change-impact analysis can be incomplete; focused regression is a conscious selection, not proof that everything else is unaffected.

6. Respond to failure without abandoning the risk model

Suppose the payment checks expose an incorrect invoice total. Diagnose the failure, establish the affected scope, and involve the responsible owner.

Do not spend the remaining window rerunning the same case until it passes. Preserve the first result and use the failure-analysis approach to distinguish product behaviour from test, data, and environment causes.

If a correction arrives, verify its deployment, retest the defect, and assess related regression needs. The earlier estimate may no longer hold. A fix that changes another shared component should trigger a revised impact assessment.

The principle from It Passed on Rerun—Is the Problem Really Fixed? also applies: retry recovery is not a substitute for understanding the original result.

7. State what the reduced suite did not establish

A useful final report distinguishes verified, failed, blocked, not selected, and not completed work.

For every significant gap, name the affected risk, available alternative evidence, owner, and next action. Do not report “all regression passed” when only the selected subset passed.

A defensible recommendation could say:

Selected checks verified the changed calculation and specified role restrictions on the candidate build. The legacy import layout remains blocked by unavailable reference data. Its compatibility is unverified; resolve the gap or explicitly decide how to handle that residual risk before releasing the affected scope.

The appropriate release authority makes the decision. QA supplies traceable evidence, an assessment, and a recommendation, consistent with the approach in Beyond Pass Rates.

8. A reusable prioritisation checklist

• Confirm the candidate changes, build, and environment.
• Map direct and shared dependencies to business journeys.
• Describe failure consequences, exposure, and uncertainty.
• Retain mandatory coverage and critical existing journeys.
• Choose checks with meaningful, independent expected results.
• Budget setup, execution, investigation, and review time.
• Replan when a blocker or late fix changes the assumptions.
• Report omitted scope and residual risks explicitly.

The goal is to obtain the most useful evidence about important release risks within the available time—and be clear about the evidence still missing.

What would you protect first if your next regression window were cut to two hours?

A rerun changes the evidence available; it does not automatically explain the original failure.

For a Principal Quality Engineer, intermittent results raise two questions: can we trust this test, and could the same condition affect a user?

This article uses a fictional search workflow to examine those questions. The timings and outcomes below are illustrative, not measurements from an employer's system.

1. What does “flaky” actually tell us?

A test that alternates between passing and failing under apparently equivalent conditions is a candidate for a flakiness investigation. “Apparently” matters: inputs, data, order, environment load, and deployed components may differ without appearing in the summary report.

Do not assume every intermittent failure is a defective test. A product race condition or an intermittently unavailable dependency can also produce variable outcomes.

In my post on five possible causes of automation failures, I used categories to guide investigation. Those categories remain hypotheses until supported by evidence.

2. Preserve both attempts

Consider this invented sequence for a test that creates a customer and searches for the new record:

A rerun that reuses the same identity can accidentally find a record left by the previous attempt. That is not evidence that a fresh creation-to-search journey succeeded.

Retain both logs, timestamps, identifiers, build details, and outcomes. Record whether a rerun started from a clean baseline. Never replace the first failure with the final pass in your investigation history.

3. Ask focused questions before changing waits

For the fictional search failure, I would investigate:

• Processing: Was creation merely accepted, or was completion confirmed?
• Search readiness: Does the documented workflow include an indexing or visibility delay?
• Locator behaviour: Did the test read the correct results container and clear old filters?
• Identity: Did it search by a unique identifier for this attempt?
• Shared data: Could another test update, delete, or claim that record?
• Order and load: Does the failure correlate with parallel execution or an earlier test?

Collect request-specific evidence. A general “service healthy” signal does not prove this request completed correctly.

If the documented workflow allows asynchronous visibility, the test needs bounded polling for the relevant state. If the product exceeds an agreed response-time requirement, extending the test timeout may conceal a performance defect.

4. Replace guesses with observable conditions

A fixed sleep assumes the system will be ready when time expires. A meaningful wait checks the state needed for the next action.

Conceptually:

create a uniquely identified record
verify the documented acceptance response
observe processing completion where the contract requires it
poll for the exact record within the agreed visibility window
verify record identity and relevant values
on timeout, report the last observed state and supporting evidence

Polling must not repeat a non-idempotent create operation. Retry observation separately from mutation, and distinguish request errors from valid “not available yet” responses.

The same principle applies to UI waits: an element being visible may not mean that business processing completed. Choose the condition that supports the test's actual claim, as discussed in meaningful passing tests.

5. Use reruns as an investigation tool

Reruns are useful when they test a hypothesis. Change one relevant condition where practical, and record it.

A pass after several simultaneous changes does not identify which change mattered. Repeating the same run until it turns green is especially weak evidence when baseline state changes between attempts.

6. Measure reliability without hiding the first result

Track first-attempt outcomes separately from results after retry. Also record the number of eligible attempts, environment, and observation period.

For example, if eight of twenty comparable first attempts fail, the observed first-attempt failure rate is 40%. If all eight reruns pass, that does not make the first-attempt rate zero.

Twenty observations are still a limited sample. The rate may change with data, concurrency, build, and environment conditions. Avoid treating one observed percentage as a permanent property of the test.

Useful measures include investigation effort, repeat failure patterns, and the number of tests with unconfirmed causes. A high retry-recovery count can indicate hidden work rather than healthy automation.

7. Quarantine with accountability

Temporarily removing an unreliable test from a release gate may be reasonable when its noise makes the gate unusable. It also removes evidence.

A quarantine record should include the affected requirement, tracking item, owner, review date, and alternative coverage. Set an exit condition that demonstrates the intended behaviour under representative conditions.

Quarantine must not become silent deletion. A critical journey still needs credible verification, and residual risk should remain visible in the release-readiness assessment.

8. Before calling the issue fixed

• Preserve the original failure and rerun evidence.
• Confirm the cause or clearly retain an unconfirmed classification.
• Verify record identity, starting state, and test independence.
• Check readiness against documented behaviour and timing expectations.
• Apply the correction to the relevant test, fixture, product, or environment.
• Exercise the condition that previously exposed the problem.
• Check representative order, concurrency, and related journeys.
• Report the observation window and remaining limitations.

No finite set of passes proves a test can never fail again. A defensible conclusion explains what was corrected, why the correction addresses the evidence, and how it was verified.

“Passed on rerun” is an observation. “Fixed” is a conclusion that needs a cause, a correction, and verification.

What evidence do you require before closing an intermittent test issue?

But did the record save? Was it saved for the correct account? Did the requested value persist after reopening the page?

The result depends on what the test actually checked. A green status can be valid for a narrow technical check while being insufficient evidence for the business behaviour described in the test title.

For a Principal Quality Engineer, that gap deserves as much attention as an obvious failure.

1. Start with the claim behind the test name

Consider a fictional test named “Update a customer's contact preference.” Its steps open a profile, select Email, click Save, and return to the list.

If those actions complete without an exception, the test demonstrates that those interactions completed. It has not necessarily established that the intended customer's preference was persisted.

A useful review question is:

What incorrect behaviour would make this test fail?

If the application silently ignores the change and the test stays green, it cannot support the claim in its title.

This connects to my earlier posts on release readiness beyond pass rates and investigating automation failures. Reliable release evidence requires both meaningful passes and well-understood failures.

All examples here are illustrative and use fictional data.

2. Move from an action to an observable outcome

For the contact-preference example, a stronger test would:

1. Identify the intended customer using a stable test identifier.
2. Establish the starting preference and choose a different permitted value.
3. Change the preference and save.
4. Check the documented success or processing outcome.
5. Reopen the record, or read it through an appropriate independent interface.
6. Confirm the customer identity and persisted preference.
7. Restore the fixture when the test's cleanup design requires it.

The baseline matters. If the preference was already Email, a broken Save operation might leave the expected value unchanged and escape detection.

Re-reading through the same stale client cache may also be weak evidence. Choose an observation point that can genuinely reveal a failure to persist, with any documented consistency delay accounted for.

3. Match each assertion to the requirement

An assertion needs a test oracle: a justified basis for deciding what the result should be. This might be an agreed requirement, a known fixture, a calculation, or a documented API contract.

Stronger does not mean checking every possible field in every test. Select assertions that support the specific requirement and expose realistic incorrect outcomes.

4. An API response can be correct at one layer and incomplete at another

Suppose a GET test checks only HTTP 200. That may be entirely appropriate for an availability check. It is inadequate if the title claims that a particular user's details are correct.

Conceptually, a functional check might look like this:

GET /users/<known-test-user-id>
assert status matches the contract
assert returned id equals the requested id
assert required fields have the documented types
assert relevant values match the known fixture

For a create or update journey, a subsequent read can test what another request observes. Use the identifier returned by creation, verify the intended values, and follow the documented consistency model.

Do not derive every expected value from the response being tested. A response compared only with itself provides little protection against incorrect behaviour.

Also distinguish request accepted, processing completed, and data persisted. Which state is promised depends on the endpoint's contract.

5. Imports need reconciliation at the right level

A completed job is evidence about the job's reported state. It is not automatically evidence that every intended business record was loaded correctly.

For a fictional import of ten parent records, each with two child rows, checking only that ten parents exist could miss all twenty child relationships.

Verify the scope relevant to the case:

• Expected record identities, not just aggregate counts.
• Important field values, effective dates, and ownership.
• Parent-child relationships and required child values.
• Per-row rejection or warning outcomes where applicable.
• Unexpected inserts or updates in the affected scope.

Reconcile against the before-state when records may already exist. A total count of ten is not proof that this run created the intended ten records.

For deliberate negative imports, an unrelated staging error does not prove the intended business validation works. Check the expected rejection condition and required absence of unintended changes.

6. Make sure the assertion controls the result

A correct assertion can become ineffective when exception handling prevents its failure from reaching the test runner.

For example, this conceptual pattern hides an unsuccessful verification:

try:
    verify_saved_value()
catch verification_error:
    log("Verification failed")
continue_to_end_of_test()

The desired behaviour is to preserve useful diagnostics and ensure the runner records the required verification as failed. Framework-specific syntax varies, but the principle does not: logging a failure is not the same as failing a test.

If using soft assertions, ensure their failures are collected and reported at the appropriate boundary. Cleanup should run as designed without erasing the original failure or silently converting it into a pass.

7. Check whether your test detects an intentional fault

In an isolated practice environment, deliberately introduce a small fault: make the saved value wrong, return the wrong identity, or remove an expected child relationship.

Then ask:

• Does the relevant test fail?
• Does it fail at the intended assertion?
• Does the message identify the missing or incorrect behaviour?
• Would a missing prerequisite cause a different, clearly explained result?

This is a focused way to challenge an assertion. Broader mutation testing can systematise similar checks, but one controlled exercise already reveals useful gaps. Do not introduce faults into shared or production environments without an explicitly approved test arrangement.

8. A short assertion-review checklist

Before accepting a passing functional test as evidence, review:

• The title states a specific, verifiable claim.
• Expected results come from a justified oracle.
• The fixture exercises the intended change or condition.
• The assertion checks the correct record, user, and relationship.
• Observation timing matches documented processing behaviour.
• Incorrect outcomes would make the test fail.
• Exceptions and soft assertions reach the test result correctly.
• Execution evidence supports the scope of the conclusion.

A passing test is useful when we can explain what it proves, how it could fail, and what it leaves unverified.

That makes green results more credible inputs to a release decision.

Which assertion have you added that caught a defect your earlier test missed?

Before choosing any of those actions, I would ask: Was an invoice ever created?

The last error in a report often describes where execution stopped, not where the problem started. A useful investigation connects the visible failure to the earlier events that made it possible.

In Beyond Pass Rates: How to Assess Whether a Release Is Ready, I discussed why release decisions need more than a green dashboard. This follow-up focuses on the evidence behind each red result.

The example below is fictional. Its timestamps, records, and messages are illustrative, not production logs.

1. Five possible causes of the same visible failure

Suppose an automated test expects a submitted charge to produce an invoice. It later times out searching for that invoice. Several explanations remain possible:

These are working categories, not mutually exclusive verdicts. One run can expose several issues. Undetermined is a valid classification until the evidence supports something more specific.

2. Reconstruct the sequence before assigning a cause

Consider this invented timeline:

The timeout is real, but it is not enough to conclude that invoice creation is defective.

The earlier rejection changes the investigation. First verify that the validation response belongs to this submission, then check the referenced fixture and the helper's success condition. Preserve timestamps and identifiers that connect those events.

3. Separate observations from hypotheses

For this example, the evidence supports these observations:

• The submission was rejected with a specific validation message.
• The helper nevertheless reported success.
• The test continued to search for an invoice.
• A report attachment failed independently.
• The subsequent search timed out.

Possible explanations still require checks:

• Is the reference genuinely inactive in the target environment?
• Does the requirement correctly reject that condition?
• Did the helper mistake completion of a click or request for business acceptance?
• Could the invoice search also have a separate locator or filter problem?

A useful record keeps confirmed findings, working hypotheses, and missing evidence separate. A plausible story is not a completed RCA.

4. Verify the suspected cause with focused checks

Assume a follow-up check confirms that the reference is inactive and that the documented rule requires an active reference. Reviewing the helper also confirms that it labels any completed submission as successful without examining the business outcome.

We can now describe two established problems: an invalid fixture for this positive case, and a test helper that incorrectly continues after rejection.

The reporting attachment error is a third issue. It should be repaired, but it did not turn the rejected charge into an accepted one.

After correcting the fixture and helper in the practice environment, repeat the relevant flow. Verify acceptance, retrieve the resulting invoice, and compare its reference, account, amount, and expected status.

If that succeeds, it supports the corrected path. It does not prove that invoice creation can never fail, or that every historic timeout had this cause.

5. Improve the test's decision points

The test should check the business outcome before entering dependent steps. Conceptually:

submit charge
inspect the business outcome
if rejected:
    record the rejection evidence
    fail this positive case
    do not run dependent invoice assertions
if accepted:
    wait for documented processing conditions
    retrieve the invoice using the expected identifier
    verify persisted business values

For a negative test, rejection may be the expected outcome. It passes only when the intended validation occurs and the required absence of unintended side effects is verified.

For asynchronous processing, acceptance and completion are separate states. A timeout should record the last observed state and relevant processing evidence. Increasing the wait is justified only when it addresses an understood timing requirement; it cannot repair invalid data or incorrect control flow.

6. Record downstream outcomes explicitly

Avoid an RCA that says only “invoice timeout.” Describe the causal chain and distinguish what each step actually did.

For this fictional case:

The positive charge submission was rejected because its required reference was inactive. The helper incorrectly reported success and continued to a dependent invoice search, which timed out. A separate screenshot-attachment error reduced report completeness. No successful charge acceptance was established in the original attempt.

For later steps, use precise outcomes: succeeded, failed, continued without verification, or not recorded. A log line saying “navigating” is not evidence that navigation completed successfully.

7. A reusable failure-triage template

Use this template for a reviewable investigation:

Different causes may need separate tracking items. Repeated symptoms with the same confirmed cause may share one primary item. Keep the mapping between runs, evidence, and tracking items explicit.

The aim of triage is to explain what the test actually proves, identify what remains unknown, and make the next action defensible.

What do you check first when a failed test ends with a timeout?

My answer would start with another question: What does that percentage leave out?

A pass rate is useful evidence, but a release decision needs context: the business journeys affected, the credibility of the tests, the scope not exercised, and the consequences of being wrong.

This article uses a fictional release to show a practical way to make that assessment. It does not describe any employer's internal systems or release process.

1. Start by checking what the number means

Suppose a planned regression suite contains 1,000 tests:

The suite has a 95% pass rate against planned tests: 950 / 1,000.

If the dashboard calculates pass rate using only executed tests, it reports 97.9%: 950 / 970.

Both calculations are arithmetically correct. They answer different questions. The second excludes 30 tests whose results are unknown.

Before interpreting a dashboard, confirm its denominator, the meaning of each status, the build and environment tested, and whether retries replaced first-attempt results. Keep pending, skipped, and blocked counts visible separately; grouping them here is only a simplified example.

2. Investigate the first meaningful failure

Twenty failed tests do not necessarily represent twenty independent product defects. A single setup failure can prevent several later checks from reaching the behaviour they were intended to verify.

For each failure, capture:

• The exact step, expected behaviour, and observed result.
• The earliest relevant error and its timestamp.
• The record, user role, build, and environment involved.
• Whether subsequent steps succeeded, failed, or were never reached.
• The evidence supporting the proposed cause and what remains uncertain.

A useful working classification is product, testware, environment, test data/setup, reporting, or undetermined.

For example, a rejected transaction followed by an invoice-search timeout should first be investigated at the rejection. The timeout may be a downstream symptom. Conversely, an exception in a screenshot attachment does not automatically mean the business transaction failed.

These classifications are investigation aids. Calling something an environment issue does not establish that the affected business journey works.

3. Map failures to business consequences

Consider two possible outcomes from the same 95% dashboard:

Release A: A critical payment journey fails for a supported user role. The failure is reproducible, and no acceptable workaround has been demonstrated.

Release B: The failed checks concern lower-impact presentation behaviour. Critical journeys have passed on the candidate build, and the remaining issues have documented workarounds and accountable owners.

The percentages are identical. The release risks differ.

For each affected journey, ask who is impacted, how often the condition occurs, whether data integrity or access controls are involved, and whether users can complete the task another way.

Keep severity, likelihood, exposure, and evidence confidence explicit. One label rarely communicates all four.

4. Treat missing coverage as uncertainty

A test that did not run supplies no evidence that its requirement passed. A test that passed with the wrong fixture may supply little useful evidence either.

Review the 30 unexecuted tests in our example:

• Which requirements and user roles do they cover?
• Are they blocked by missing data, an unavailable environment, or unfinished automation?
• Is there independent API, manual, or integration evidence for the same behaviour?
• Does that alternative evidence apply to the candidate build and relevant configuration?

Alternative evidence can reduce uncertainty, but it should be traceable and genuinely test the requirement. A mocked success response is not equivalent to checking that a real record persisted correctly.

A simple coverage matrix connecting critical journeys to requirements, roles, test results, and evidence gaps is often more useful than a larger aggregate percentage.

5. Evaluate retries and fixes honestly

A successful rerun is evidence that a test passed on that attempt. It does not, by itself, explain why the earlier run failed.

Retain first-attempt outcomes and investigate repeated intermittency. Otherwise, repeatedly retrying until green can hide reliability problems.

Similarly, keep these states distinct:

Fix proposed → fix merged → fix deployed → targeted verification passed → affected regression checked.

A merged change is not evidence that the candidate environment contains it. A targeted test passing does not automatically establish that related journeys remain unaffected.

The investigation should show which state has actually been reached, with supporting build or deployment information where available.

6. Make the recommendation actionable

A Principal Quality Engineer should translate technical findings into a decision others can assess.

A fictional release review might contain:

Possible recommendations include holding the release, reducing scope, or proceeding with explicitly accepted residual risk. The appropriate release authority owns the final decision; QA provides the evidence, recommendation, and unresolved questions.

Avoid broad statements such as “safe to release” when the evidence only supports a narrower conclusion.

7. A checklist for the next release review

Before recommending a release, check:

• The pass-rate denominator and status definitions are understood.
• The evidence applies to the candidate build and relevant environment.
• Critical business journeys and supported roles have credible coverage.
• Failures are investigated, with primary causes separated from downstream symptoms.
• Pending, skipped, and blocked work is visible, with its impact explained.
• Reruns and intermittent failures remain traceable.
• Required fixes are deployed and verified, with appropriate regression evidence.
• Residual risks have owners, mitigation, and explicit acceptance where required.
• Monitoring and rollback arrangements are understood for the affected scope.

The question I want a release review to answer is: What can we confidently say works, what remains uncertain, and who is accepting the remaining risk?

That gives a pass rate its proper context and makes the recommendation useful.

What evidence has changed your release recommendation even when the dashboard looked green?