Flaky Tests: Four Causes and Their Artifacts

A flaky test is not a test that fails sometimes; it is a test whose failure you have not yet attributed. Retrying until it passes produces a green run and destroys the evidence that would have told you which of four causes was at work. BotBrowser documents per-launch and per-context storage isolation, which removes one mechanism behind the shared-state cause rather than the cause in general. It does not fix timing, network or test-design causes, and it does not make a suite deterministic by itself.

TL;DR

• Capture the first failing attempt before any retry. A retry that overwrites it has removed the only evidence you had.
• Attribute the failure to one of four causes: shared browser state, timing, network or environment drift. Each leaves a different artifact.
• Two failures with the same signature is a reproducible observation. It narrows the cause but does not prove the failure is not intermittent.
• Keep retries as a recording device. Record every attempt, then fix the cause and remove the retry from that test.
• BotBrowser removes one mechanism of shared state by isolating storage per context or launch. It does not fix the other three causes.

Contents

• Four causes and the artifact for each
• Why a retry erases the evidence
• A retry policy that records
• A state-isolation checklist
• Practical conclusion

Four causes and the artifact for each

The useful first question is not "how often does it fail" but "what does the failure look like when it happens". Each of the four causes leaves a signature, and each signature needs a specific artifact to confirm it. An artifact proposes a hypothesis; confirming it takes a control comparison, and collecting the wrong artifact produces a confident and wrong attribution.

Cause	Signature	Artifact that confirms it
Shared browser state	Fails only when another test ran first	Storage, cookie and cache dump before the failing step
Timing	Passes with a longer timeout, fails with the normal one	Trace with action timings and the event it waited for
Network	Fails on the same request rather than the same step	Request log with status codes and retry count
Environment drift	Fails on one runner, profile or viewport only	Launch arguments, profile hash and viewport for both runs

The state cause is the one a browser suite under-reports, because the artifact is invisible in the failure message. The test-data and browser-state hygiene practice covers how to keep that state from accumulating in the first place. A test that fails only after another test has run usually fails on an assertion about content, and the message points at the assertion rather than at the leftover storage. Dumping storage and cookies immediately before the failing step is what turns that into an attribution.

The timing cause is the one most often "fixed" without being diagnosed, and the waits and stable-test analysis covers what a correct wait is waiting for. A test that passes with a longer timeout has demonstrated that the wait was too short, which is useful evidence and not a repair. The trace shows whether the element appeared late, whether the network was slow, or whether the action fired before the page settled, and those three answers lead to different fixes.

The network cause and the timing cause are easy to confuse because both appear as "the element was not there". The distinction is that a network cause reproduces on the same request while a timing cause reproduces on the same step with a different request. Logging the request status and the retry count separates them without guessing.

The environment cause is the one that looks like a framework bug. A test that fails on one runner and passes on another is usually failing because the two runs had different launch arguments, different profile values or a different viewport. Recording all three for both runs is cheap, and it is the artifact that makes the comparison possible.

A fifth signature deserves a note because it is frequently misread as one of the four: the test depends on data that changes over time. A price assertion that compares against a literal, a date assertion that assumes today, or a list assertion that assumes a fixed count all fail on a schedule rather than intermittently. The artifact is the input data and its timestamp, and the fix belongs in the fixture rather than in the browser. Treating it as a state or timing cause leads to retries that pass on some days and fail on others, which is the most expensive possible outcome.

Order of investigation matters as much as the table. Start with the cause that is cheapest to rule out and cheapest to fix, which is almost always state, because a fresh context either reproduces the failure or it does not. Then check environment, because comparing two recorded launch configurations is a diff rather than an experiment. Timing and network come last, because diagnosing either requires a trace and a request log that cost real time to produce.

There is also a hygiene point that applies to all four. An artifact is only useful if it was captured at the moment of failure rather than reconstructed later. A screenshot taken after the test framework already navigated away, a storage dump taken after the cleanup hook ran, or a request log filtered to errors only will each look like an artifact while containing none of the evidence. Capture at the failure hook, before any teardown, and record which hook produced the artifact so a later reader can tell whether the timing was right.

Why a retry erases the evidence

Playwright documents retries as a test-runner feature that re-runs failed tests, and documents fixtures and parallel execution as separate features with their own scoping rules. None of those documents claims that retries make a suite reliable. The feature re-runs a test; what happens to the evidence of the first attempt depends entirely on how the suite is configured to record it.

The default failure mode is subtle, and it is visible in one field: status=passed with retries=1. Playwright reports a retried test that eventually passed as flaky, with testInfo.status equal to passed, so a report that reads only the final status field sees a pass and nothing else. The stronger problem is that the first attempt often has no trace at all: the default trace: 'off' records nothing, and on-first-retry starts recording only when a retry begins. The evidence is missing rather than overwritten, unless a custom output path reuses one filename across attempts.

That is why a retry is a recording device rather than a fix. Used well, it tells you which tests are unstable and gives you a second observation of the same failure. Used badly, it converts a real defect into a green build and moves the discovery to production, where the artifact no longer exists.

There is also a statistical trap. A test that fails on one run in ten will pass a single retry nine times in ten, so a suite with retries enabled looks far more stable than it is. The failure rate has not changed; only the reporting has. Two failures in a row with the same signature is the strongest signal available, but it is still a hypothesis: a network cause can produce an identical signature twice and remain intermittent, so the attribution needs a control comparison.

A retry policy that records

A policy that records is small: keep every attempt, name the artifact per attempt, and refuse to summarise away the first failure. The builder below is a constructed fixture showing what the record should contain.

// Constructed fixture: three attempts of one test, two failures with a matching artifact prefix.
const attempts = [
  { n: 1, status: 'failed', artifact: 'storage-dump:t1.json', at: '2026-11-10T09:00:12Z' },
  { n: 2, status: 'failed', artifact: 'storage-dump:t2.json', at: '2026-11-10T09:00:48Z' },
  { n: 3, status: 'passed', artifact: 'trace:t3.zip', at: '2026-11-10T09:01:30Z' },
];

const failures = attempts.filter(a => a.status === 'failed');
const sameArtifact = failures.length > 1 && failures[0].artifact.split(':')[0] === failures[1].artifact.split(':')[0];
const verdict = sameArtifact ? 'repeated_signature' : failures.length ? 'single_failure' : 'clean';

const lines = [
  `attempts=${attempts.length} failures=${failures.length}`,
  ...attempts.map(a => `attempt=${a.n} status=${a.status} artifact=${a.artifact} at=${a.at}`),
  `verdict=${verdict}`,
];
console.log(lines.join('\n'));

The constructed output below is the point: the third attempt passed, and the record still says the signature repeated.

attempts=3 failures=2
attempt=1 status=failed artifact=storage-dump:t1.json at=2026-11-10T09:00:12Z
attempt=2 status=failed artifact=storage-dump:t2.json at=2026-11-10T09:00:48Z
attempt=3 status=passed artifact=trace:t3.zip at=2026-11-10T09:01:30Z
verdict=repeated_signature

Three rules make the policy work in practice. Name the artifact by cause rather than by file, so storage-dump and trace are distinguishable at a glance. Keep the attempts in order with their timestamps, so a reviewer can see whether the failure was immediate or delayed. And make the verdict a derived field rather than a human judgement, so the report cannot quietly call a repeated failure a flake.

Two more rules belong to the report rather than to the builder. First, publish the failure count next to the pass count, because a suite that reports only final statuses hides the failures that were retried away. Second, keep the artifact from every attempt rather than only the last, because the first attempt ran before any retry-time cleanup, and a suite that clears state in a retry hook has already changed the conditions that produced the failure.

Retry scope is a third consideration. A retry configured at the test level gives you repeated observations of one test; a retry configured at the file level reruns only failed tests, while a retry configured for a serial group reruns the whole group including tests that did not fail, which changes which tests share state and can make the second attempt a different experiment. Playwright also documents a group or serial-group retry, where every test in the group is retried together, so the second attempt runs a different set of tests. Prefer the narrowest scope that reproduces the failure, and record the scope in the attempt record so a later reader knows what was actually re-run.

A retry policy also needs an owner and an exit condition. The owner is whoever can fix the cause, and the exit condition is a date by which the retry must be removed or the test quarantined. Without both, a retry added during an incident becomes permanent infrastructure that no one revisits, and the suite slowly fills with tests whose real status is unknown.

Quarantine is the honest alternative to retrying forever. Moving a known-broken test out of the required set, with its artifact attached and an owner named, keeps the suite's signal intact while the cause is investigated. It is a visible failure rather than a hidden one, and visible failures get fixed.

The policy also needs a stopping rule. If two attempts fail with the same signature, stop retrying and change one variable instead: the signature is reproducible, so another attempt under the same configuration adds no information. The next step is the control comparison from the table, not a third retry. Continuing to retry after that point spends time producing evidence that already exists, and it moves the suite toward the green-by-retry state the policy exists to prevent.

A state-isolation checklist

The state cause is worth checking first because it is the cheapest to remove and the easiest to mistake for something else. Each item below produces an observation rather than a belief, so the checklist can be run before any diagnosis of the other three causes.

Check	Observation it produces
Fresh context per test	Whether the failure survives an empty context
Storage cleared before navigation	Whether leftover local or session storage mattered
Cookies cleared, not just expired	Whether a persisted cookie changed the outcome
No shared profile across workers	Whether two workers wrote to one profile
Fixed viewport and device values	Whether layout differences changed the result
Recorded launch arguments per run	Whether two runs were configured identically

The fixture and isolation model describes the scoping that makes a fresh context per test practical. The first row is the decisive one. If a test fails in a shared context and passes in a fresh one, the cause is state, and no amount of timeout adjustment will address it. If it fails in both, the state cause is unlikely and the next two causes deserve the effort.

The fourth row is the one that parallel execution makes urgent. Playwright documents parallel execution as running tests in separate workers, and a worker that reuses one profile across tests can leave state behind for the next test in that worker while leaving other workers clean. The result is a failure pattern that looks random by worker count rather than by test.

Worker reuse produces a second, subtler signal: the failure moves. If the same test passes in a serial run and fails only when the suite runs in parallel, and the failing assertion changes between runs, the cause is almost certainly shared state crossing a worker boundary. Recording the worker identifier with each attempt turns that from a suspicion into a pattern, because the failing attempts will cluster on one worker rather than spreading evenly.

Clearing storage is not the same as clearing cookies, and neither is the same as starting a fresh context. Local storage, session storage, IndexedDB, the service worker cache and the HTTP cache each survive different resets, and a test that depends on one of them will keep failing after the others are cleared. The cheapest complete reset is a new browser context per test, which is also the observation the first checklist row produces.

Service workers deserve their own line because they survive a page reload and can serve a stale response long after the storage the test thinks it cleared. A test that fails on the first run after a deployment and passes afterwards is often reading a cached response from a worker registered by an earlier test. Recording whether a service worker was registered at failure time is the artifact that separates this from a network cause.

The last checklist item is the one that makes the other five reproducible. Two runs with different launch arguments are not the same experiment, and a failure that appears only in one of them may be a configuration difference rather than a code difference. Recording the arguments, the profile hash and the viewport together gives a reviewer a single comparable string, and comparing two of those strings is a diff rather than a debate.

BotBrowser's profile-management documentation states that each launch uses its own user-data-dir, cookies and session settings, and that inside one process storage, cookies and context settings are isolated by the BrowserContext. Its profile-configuration documentation lists the environment inputs a profile carries, such as locale, timezone, screen and WebGL. Together those remove one mechanism of cross-test state: a fresh context or a fresh launch cannot inherit the previous test's storage. That is narrower than removing the state cause in general, and it says nothing about timing, network or test design, so a suite with a timing problem stays flaky with a perfect profile.

FAQ

Is a test that passes on retry a flaky test?

It is a test with an unattributed failure. The retry changed the outcome, not the cause. Keep the first attempt's artifact and attribute the failure before deciding the test is healthy.

Can I remove flakiness by increasing timeouts?

Only if the cause is timing, and only after the trace shows that the wait was the limiting factor. Increasing a timeout for a state or network cause hides the failure for longer without changing it.

Why does a test fail only on one runner?

That signature points at environment drift: different launch arguments, a different profile or a different viewport. Record all three for both runs and compare them before looking for a framework bug.

Does BotBrowser make a flaky suite deterministic?

No. It documents per-launch and per-context storage isolation, which removes one mechanism behind the shared-state cause. Timing, network and test-design causes remain, and no browser configuration makes a suite deterministic by itself.

Sources

• Playwright test retries
• Playwright test fixtures
• Playwright parallel execution
• BotBrowser profile management
• BotBrowser profile configuration
• BotBrowser automation consistency

Practical conclusion

Capture the first failing attempt before any retry, attribute the failure to state, timing, network or environment using the artifact that separates them, and only then change one thing. Keep retries as a recording device: every attempt with its artifact and timestamp, a derived verdict, and a stopping rule once two attempts fail the same way. Test the shared-state mechanism first because it is cheapest to check, and remember that per-context storage isolation addresses only that mechanism.