flaky test
A test that sometimes passes and sometimes fails on the same code, because it depends on something it does not control.
In more detail
Common causes are time, run order, randomness, networks and shared state. Retrying hides the evidence and slows the build; instead reproduce the failure, name the moving input and make it explicit or isolated. Running the test repeatedly, alone and in a shuffled suite, helps find the cause.
Example
Visual tests are flaky when a screenshot catches an animation mid-way, a font still loading or different text rendering; freeze animations, await fonts, fix the viewport and run in one container.Related terms
- visual regression testing — Testing that takes screenshots of components and compares them pixel by pixel with approved baseline images to catch unintended visual changes.
- deterministic — Always producing the same result from the same inputs, with nothing left to chance or timing.
- web font — A font file downloaded by the page, declared with @font-face, instead of one already installed on the user's device.
- test suite — A collection of tests run together to check a piece of software.
- web-first assertion — A Playwright assertion that keeps rechecking the page until the condition is true or a timeout expires.
- network mocking — Intercepting a test’s outgoing network requests and answering them with fixed fake responses.
- end-to-end test — A test that drives the whole system from the outside, often through a browser or public API, with every part running.
- retry — Sending a request again after it failed or timed out, in the hope that it succeeds the second time.
- test isolation — The property that each test sets up everything it needs and leaves nothing behind, so tests pass in any order.
- frozen time — Making the current time a fixed value during a test, by passing an instant in or patching the clock.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
- Visual Regression Testing: Playwright & Snapshot Differencing — Design Systems & Component Engineering
- Testing Strategies And The Test Pyramid — DevOps & Site Reliability Engineering
- Integration Testing with WebApplicationFactory — Modern .NET Core, C# & Enterprise Microservices
- End-to-End Testing with Playwright: Multi-Tier Suites — Production Full-Stack Engineering
- Fixtures And Their Scope — Automated Testing, TDD & Quality Engineering
- Testing Time And Randomness — Automated Testing, TDD & Quality Engineering
- Flaky Tests And How To Kill Them — Automated Testing, TDD & Quality Engineering
References
- Playwright: visual comparisons
- Google Testing Blog: Flaky Tests at Google and How We Mitigate Them
- Martin Fowler: Eradicating Non-Determinism in Tests
- Unit testing best practices
- Playwright: Best practices
- pytest: flaky tests