Flaky tests in CI

A flaky test passes and fails on the same code. It is worse than a failing test: people stop trusting red builds, re-run until green, and one day ignore a real failure. This guide gives you a way to find flaky tests, fix them, and stop new ones. The examples use Playwright; the ideas apply to Selenium and any other tool.

Step 1: confirm it is flaky

Run the single test many times in a row:

npx playwright test tests/checkout.spec.ts -g "places an order" --repeat-each=20

If it passes 20 times, run it as CI does: with the other tests, in parallel, on a slower machine (a container with limited CPU). Many flaky tests only show up under load or when run next to a test that shares data. Then try --workers=1: if the failures vanish, tests are interfering with each other through shared state.

Step 2: find the flaky ones you do not know about yet

Let CI retry failed tests (retries: 2 in the Playwright config) and count the tests that failed and then passed. Playwright marks those as flaky in its JSON report. This small script lists them:

// flaky-report.mjs   usage: node flaky-report.mjs reports/results.json
import fs from 'node:fs';

const report = JSON.parse(fs.readFileSync(process.argv[2], 'utf8'));
const flaky = [];
const failed = [];

function walk(suite, trail = []) {
  const here = suite.title ? [...trail, suite.title] : trail;
  for (const child of suite.suites ?? []) walk(child, here);
  for (const spec of suite.specs ?? []) {
    for (const t of spec.tests) {
      const name = [...here, spec.title].join(' > ');
      if (t.status === 'flaky') flaky.push({ name, attempts: t.results.length });
      if (t.status === 'unexpected') failed.push(name);
    }
  }
}
report.suites.forEach((s) => walk(s));

console.log(`${flaky.length} flaky, ${failed.length} failed`);
for (const f of flaky) console.log(`  FLAKY (${f.attempts} attempts): ${f.name}`);
for (const f of failed) console.log(`  FAILED: ${f}`);
process.exit(failed.length ? 1 : 0);

Run it after every CI build (add ['json', { outputFile: 'reports/results.json' }] to the reporters) and keep the output. A test that shows up three times in a week is your first target.

Step 3: find the cause

Nearly every flaky test comes from one of five causes.

1. A fixed sleep instead of waiting for a condition

A pause of two seconds is too short on a slow machine and wastes time on a fast one.

// Before: guesses how long the page needs
await page.getByRole('button', { name: 'Save' }).click();
await page.waitForTimeout(2000);
expect(await page.locator('.toast').innerText()).toBe('Saved');

// After: waits until the toast says "Saved", and gives up after the timeout
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.locator('.toast')).toHaveText('Saved');

Playwright's expect(locator) assertions retry until true. In Selenium use an explicit wait on a condition. See the Selenium wait questions in Selenium interview questions.

2. Tests that share data

Two tests use the same account or record, and the order or timing decides who wins.

// Before: every test logs in as the same user and edits the same profile
await login(page, '[email protected]');

// After: each test creates its own user through the API
const user = await api.createUser({ email: `qa.${Date.now()}.${testInfo.workerIndex}@example.com` });
await login(page, user.email);

Create what the test needs, use it, and remove it. Never rely on data another test left behind.

3. Tests that depend on order

A test passes only because an earlier test ran first. Run the file with --shard or in random order and it breaks. Every test must set up its own start state. If a flow genuinely has steps, put them inside one test, not across tests.

4. A slow or different environment

The test passes on a quick laptop and fails on a busy CI machine because a request takes longer than an implicit assumption. Check the trace: where did the time go? Increase the right timeout only after you have proved the wait is for a condition that really takes that long; otherwise fix the wait. Also check the viewport, locale and time zone: CI machines often differ.

5. Not waiting for the right network call

The page shows old data until a request finishes, and the test reads it too early.

// After: wait for the request that matters while doing the action that causes it
const [response] = await Promise.all([
  page.waitForResponse((r) => r.url().includes('/api/orders') && r.status() === 201),
  page.getByRole('button', { name: 'Place order' }).click(),
]);
await expect(page.getByTestId('order-confirmation')).toBeVisible();

Avoid waiting for "network idle": pages with polling or analytics may never be idle.

Step 4: use the trace, not guesswork

Set trace: 'retain-on-failure' (or 'on-first-retry') in the config and upload test-results as a CI artifact. Open the trace of a flaky failure and look at the page at the moment the action failed: was the element covered? still loading? already gone? It turns "it just fails sometimes" into a specific cause. The GitHub Actions tutorial shows how to save it.

Step 5: quarantine, own and fix

While a flaky test waits for its fix it should not block the whole team. A simple process:

1. Tag it in the title: test('places an order @quarantine', ...).
2. Run the main suite without it: npx playwright test --grep-invert @quarantine. Run the quarantined ones separately, so you still see whether they pass: npx playwright test --grep @quarantine.
3. Open a ticket with an owner and a date. A test that stays quarantined for more than a couple of sprints should be fixed or deleted.
4. When it is fixed, remove the tag and watch it for a week of CI runs.

Do not quarantine silently. The number of quarantined tests belongs on the team's dashboard.

Step 6: keep new ones out

• Review tests for fixed sleeps, shared accounts and copied locators.
• Run new tests many times before merging (--repeat-each=10).
• Track the flake rate from the report script, and watch the trend.
• Treat a flaky test as a bug: it has an owner, a priority and a root cause written down.
• Keep tests independent, so they can run in any order and in parallel.

Try it yourself

1. Write a test that passes only on its third try (testInfo.retry >= 2), run it with retries: 2 and the JSON reporter, and run the script on the result.
2. Take one of your own tests with a waitForTimeout and replace it with a condition.
3. Run your suite with --workers=1 and then with several workers and compare the failures.

For the matching interview answers, see the CI/CD questions (question 9) and the Playwright questions (question 29).