Flaky Tests: Why Tests Fail Randomly and How to Fix Them

The Quick Answer

A flaky test is a test that passes and fails on the same code. Nothing changed in the application, nothing changed in the test, and yet one run is green and the next is red. Every team that automates testing eventually meets them, and every team underestimates how much damage they do.

The core problem: A flaky test destroys the one thing a test suite exists to provide - a trustworthy answer. Once engineers learn that red does not reliably mean broken, they start re-running failed builds instead of reading them, and the suite stops being a signal.

The short version of the fix: Measure flakiness as a number, quarantine unreliable tests so they stop blocking releases, fix the root cause rather than adding retries, and hold new tests to standards that prevent flakiness from entering the suite in the first place.

This article covers what actually causes tests to behave non-deterministically, how to detect and measure it, what to do with a flaky test the moment you find one, and how to fix the underlying causes instead of masking them.

What Makes a Test Flaky

A test is flaky when its result is not determined solely by the code under test. Something else influences the outcome - timing, shared state, execution order, the environment, an external service - and that something varies between runs. The test is not wrong about the application being broken; it is unable to give a consistent answer at all.

This is different from a genuinely failing test and different from a badly written one. A failing test reports a real defect and keeps reporting it. A badly written test may assert the wrong thing but does so consistently. A flaky test is unpredictable, which is what makes it so corrosive: you cannot tell from a single result whether you have found a bug or hit noise.

Intermittent Does Not Mean Harmless

Teams often classify flaky tests as a low-priority annoyance because each individual failure resolves itself on re-run. The cost is not in the individual failure - it is in what the team learns from it. After a few weeks of intermittent red builds, the reflex becomes "just run it again," and that reflex applies to real failures too. A flaky suite trains engineers to ignore exactly the signal you built it to produce.

Flakiness Sometimes Points at a Real Bug

An important nuance: a flaky test can be reporting a genuine race condition in the product. If a test fails because the application occasionally renders a stale value or occasionally returns before a write completes, the non-determinism is in the product, not the test. Before you dismiss any flaky test, ask whether the application itself is the unstable party. Some of the most valuable bugs a team ever ships a fix for start out looking like test noise.

The Real Cost of Flaky Tests

Flakiness is expensive in ways that are easy to miss because the cost is distributed across the team rather than concentrated in one visible failure.

• Lost trust in the suite. This is the largest cost by far. When failures are routinely dismissed, the suite no longer gates anything, and real regressions reach production through a pipeline that was supposed to catch them.
• Wasted engineering hours. Every re-run costs pipeline minutes, and every investigation costs an engineer context switching into a failure that turns out to be nothing.
• Slower releases. A pipeline that has to be run two or three times before it goes green stretches the time between a merge and a deployable build.
• Distorted quality metrics. Pass rates, defect counts, and coverage reporting all become unreliable when a portion of the failures are noise. Teams end up making decisions on numbers that do not describe reality.
• Reduced appetite for automation. Once a team has been burned by an unreliable suite, proposals to automate more get met with resistance - which is the opposite of what the team needs.

The compounding effect matters most. A suite with a handful of flaky tests is an annoyance. A suite where any given run has a meaningful chance of failing for no reason is functionally useless as a release gate, and getting back from that state takes far more effort than preventing it would have.

Common Causes of Flaky Tests

Most flakiness traces back to a small number of recurring causes. Recognizing the pattern is usually most of the diagnosis.

Two of these deserve particular attention because they are the most common. Timing is the single biggest source of flakiness in end-to-end tests: a fixed wait that works on a developer's laptop fails on a loaded CI runner. Shared state is the biggest source in integration suites, and it usually appears the day a team turns on parallel execution, because tests that quietly depended on running alone suddenly run alongside each other.

How to Detect and Measure Flakiness

You cannot manage flakiness by impression. "The suite feels unreliable lately" is not something a team can act on, and it is not something that tells you whether last month's cleanup worked. Flakiness needs a number.

The Flake Rate

The most useful single metric is the flake rate: the share of test runs that produced a different result on the same code. Measure it per test and for the suite as a whole. A test that fails one run in twenty is a flaky test even though it looks green almost all the time, and it will show up as a mystery failure roughly once a week on an active pipeline.

Track a small set of numbers over time rather than one snapshot:

• Per-test flake rate - which specific tests are unreliable, ranked worst first
• Suite flake rate - the probability that a full run fails for no real reason
• Re-run frequency - how often engineers manually re-trigger a build, which is a direct proxy for lost trust
• Time to fix - how long a test stays in quarantine before it is repaired or deleted

How to Surface Them

Flaky tests hide unless you look for them deliberately. Three practices surface them quickly. First, keep historical results rather than only the latest run, so you can see a test's pattern over dozens of executions instead of one outcome. Second, re-run the suite against an unchanged commit on a schedule - if the code did not change, something else did, so any failure there is worth investigating. Check the pattern before you label it: a test that alternates between passing and failing across those runs is flaky, while one that has started failing every time usually points at an expired credential, a drifted dependency version, or a broken environment, and that is a real problem rather than noise. Third, run the suite in a different order and with different parallelism than usual, which exposes order dependencies that a fixed pipeline configuration hides.

All three depend on having result history you can actually query. If your test results live only in pipeline logs that roll off after a week, flakiness is invisible by construction. Keeping runs and their outcomes in one place - through test reporting that retains history per test case - is what turns "this fails sometimes" into a ranked list you can work through.

Quarantine: What to Do the Moment You Find One

When you identify a flaky test, you have three options, and the wrong default is the most common one.

The wrong default: Leave it in the pipeline and re-run the build when it fails. This keeps the noise, keeps the erosion of trust, and guarantees the test is never fixed, because nothing forces the issue.

The better approach is an explicit quarantine. Move the test out of the blocking suite so it stops gating releases, but keep it running and keep recording its results. Quarantine is not deletion and it is not a hiding place - it is a holding area with rules attached.

Rules That Make Quarantine Work

• Every quarantined test has an owner. An unowned quarantine list grows forever.
• Every quarantined test has a deadline. Two weeks is a reasonable default. At the deadline it is fixed or deleted - no third option.
• Quarantine stays visible. Report its size alongside your pass rate. A quarantine list nobody sees becomes a graveyard.
• Cap the list. If quarantine exceeds a small percentage of the suite, stop adding tests and spend a cycle clearing it.
• Deleting is a legitimate outcome. A test that is unreliable and not worth the effort to stabilize is worth less than nothing, because it costs attention while providing no signal.

The point of quarantine is to separate two decisions that teams usually make at once and badly: "should this block the release right now" and "is this test worth keeping." The first needs an answer in seconds. The second deserves a considered one.

Fixing the Root Causes

Once a test is quarantined, the actual repair work follows the cause. Four patterns cover the majority of cases.

Replace Time With Conditions

The most common fix is to stop waiting for an amount of time and start waiting for a state. A fixed pause encodes an assumption about how fast the environment is, and that assumption breaks on a loaded runner. Wait for the specific thing you need - an element to be interactive, a request to complete, a record to reach a status - with a generous timeout and a clear failure message. This single change removes a large share of end-to-end flakiness.

Make Every Test Independent

Each test should create the state it needs and clean up after itself, so it produces the same result whether it runs first, last, alone, or in parallel with fifty others. That means no reliance on data another test created, no reliance on execution order, and no shared mutable fixtures. This is more setup code, and it is worth it. Independent tests can also be parallelized safely, which usually pays back the cost in pipeline time.

Control the Inputs

Anything that varies between runs and is not the subject of the test should be pinned down: inject a fixed clock rather than reading the system time, seed random generators, and stub external services for functional tests. Real integrations still need testing, but that belongs in a separate suite where an outage means "the integration is down" rather than "the feature is broken."

Isolate the Data

Give each test its own data rather than sharing a fixture set that every test reads and some tests modify. Isolated data removes an entire category of interference, and it makes failures far easier to diagnose because the state a test ran against is knowable. This is closely tied to broader test case management discipline: when preconditions are documented as part of the test case rather than assumed, the dependency on shared state becomes visible before it becomes a flake.

Retry Logic: When It Helps and When It Hides Bugs

Automatic retries are the most tempting and most misused response to flakiness. A retry turns a red build green without anyone having to understand why it was red - which is exactly the problem.

The rule: Retries are acceptable as a way to reduce noise while you fix the cause. They are not acceptable as the fix. A retry policy with no accompanying repair work converts a visible problem into an invisible one, and an invisible flaky test can mask a real intermittent product bug indefinitely.

If you do use retries, constrain them. Allow one retry, not three - a test that needs three attempts is not flaky, it is broken. Always record that a retry happened and count it against the test's flake rate, so retried tests surface in reporting instead of disappearing into a green checkmark. Never retry at the suite level; retry the individual test, so one unreliable test does not conceal a genuine failure elsewhere in the same run. And treat any test that is retried repeatedly as a quarantine candidate rather than a permanent resident of the pipeline.

The distinction that matters: a retry that is measured is a tolerable stopgap, and a retry that is silent is a way of deceiving yourself about the state of your test suite.

Preventing Flakiness in New Tests

Cleaning up an unreliable suite is far more expensive than not building one. A few standards applied at review time keep most flakiness out.

• No fixed sleeps. Make this a review rule with no exceptions; every wait must name the condition it is waiting for.
• No dependence on execution order. New tests must pass when run alone and when run in a shuffled suite.
• Own your data. A test creates what it needs and removes it afterwards.
• Stable selectors only. Target attributes that exist for testing, not positions or generated names.
• No unpinned randomness or system time. If a test's behavior can vary by input it does not control, it will eventually vary.
• Run new tests repeatedly before merging. Executing a new test twenty times in CI catches most latent flakiness before it reaches the main suite.

These standards work best when they are part of how test cases are written rather than a checklist applied afterwards. Writing explicit preconditions and expected states into the test case itself makes the assumptions visible, and AI-assisted test case creation can help produce that level of detail consistently instead of leaving setup implicit. The broader habits that keep suites healthy are covered in our guide to test automation best practices.

Conclusion

Flaky tests are not a cosmetic problem with a test suite - they are an attack on the reason the suite exists. A test that gives different answers about the same code cannot gate a release, and a team that has learned to re-run instead of investigate has lost the safety net it paid for.

The path out is straightforward, if not effortless. Measure flakiness as a rate rather than describing it as a feeling. Quarantine unreliable tests immediately so they stop blocking work, with an owner and a deadline attached to each one. Fix the underlying cause - almost always timing, shared state, uncontrolled inputs, or a genuine race condition in the product. Use retries only as a temporary, measured stopgap. And hold new tests to standards that keep the problem from coming back.

Doing that requires seeing your results over time rather than one run at a time. QA Sphere keeps test case history, test run results, and reporting in one place, so a test that fails one run in twenty shows up as a pattern instead of a mystery. See pricing or book a demo to see how it fits your pipeline.