The 40-Minute Feedback Loop: Auditing a Test Suite That's Quietly Eating Your Sprint
Photo: Alexander P Kapp, CC BY-SA 2.0, via Wikimedia Commons
Every team has a number they don't say out loud. It's the gap between typing git commit and knowing whether the thing you just wrote actually works. For a lot of teams that number sits somewhere north of forty minutes, and nobody remembers approving it.
The worst part isn't the runtime. It's that the runtime punishes the exact behavior you want. The developer who runs the suite locally before pushing — the conscientious one — pays the full tax on every change. The developer who skips it and lets CI sort it out gets feedback much cheaper. That's a perverse incentive you almost certainly did not design on purpose.
The cost isn't wall-clock time, it's the context switch
A forty-minute suite doesn't cost forty minutes. You kick off the run, tab into Slack, get pulled into a thread, start reading someone else's PR, and drift back well after the tests finished. Effective cost per run is closer to an hour, and the reload penalty when you return to your editor is the expensive half.
Do the math for a team of eight running tests three times a day. That's a real slice of a sprint that never appears on a planning board. Nobody files a ticket for it, which is exactly why it survives.
Measure before you optimize anything
You can't fix a suite you haven't profiled. Most runners will tell you where the time goes if you ask:
- Jest:
--verboseplus the built-in timing report, or run with--detectOpenHandlesif you suspect hung resources - pytest:
--durations=25gives you the slowest tests per run - RSpec:
--profile 25 - Go:
go test -v ./...and look for the slow packages, not slow tests - Vitest: the reporter output includes per-file timings by default
The shape is almost always the same. Ten to twenty tests account for the majority of the runtime, and a chunk of them are integration tests wearing a unit test costume. Look for beforeEach hooks that spin up a database, seed fixtures, or hit a container. That's your first cut.
Draw the unit/integration line honestly
Most slow suites are slow because the boundary was never defined. Draw it in writing:
- Unit tests run in-process. No network, no filesystem writes, no real clock, no container. Under 50ms each as a rule of thumb.
- Integration tests hit real services: Postgres, Redis, S3, a stubbed HTTP server, whatever the code actually talks to.
Then tag them. Unit tests run on every save and every commit. Integration runs on push. The full suite including end-to-end runs on PR and nightly. Developers get a fast local loop, and CI still does the thorough thing before anything merges.
If half your suite is mislabeled, you don't have a slow suite. You have a mislabeled integration suite and a unit suite that doesn't exist yet.
Quarantine flaky tests without sentiment
Flaky tests destroy trust faster than slowness ever will. Once a developer sees a red run turn green on retry with zero code changes, red stops meaning broken. At that point the suite is decoration.
Set a policy and let a bot enforce it. Any test that fails and then passes on retry three times in thirty days goes into quarantine automatically. Quarantined tests still run — nightly, in a job nobody blocks on — and they carry a ticket and an owner with a two-week deadline. Fix or delete. No limbo, no skip comment from two years ago guarding a test that no longer asserts anything meaningful.
Parallelize and shard, but watch your shared state
Once the suite is honestly split, parallelism gets much simpler:
- Shard by historical timing data, slowest first, not alphabetically
- Use your CI provider's matrix (
--shard=2/6,pytest-split, Knapsack) so each node gets roughly equal work - Separate database schemas or containers per shard so tests don't collide
- Randomize execution order with a fixed seed — this catches order-dependence, which is a leading cause of flakes you'll otherwise chase for months
If your suite gets slower when you add parallel workers, you have shared global state. Find it before you add more hardware.
Cache aggressively, but honestly
Dependency installs are usually the second-largest cost after the tests themselves. Cache the lockfile install, the build output, and the Docker layers. Remote build caches (Turborepo, Nx, Bazel, Gradle's build cache) can eliminate whole categories of work on unchanged packages.
One warning: never cache results keyed by something that can silently miss. A cache that returns green from a stale key is more dangerous than a slow suite, because it manufactures confidence. If you can't verify the key covers env vars, tool versions, and config, don't cache the result — cache the setup instead.
A realistic target
Ten minutes is the number worth aiming at, and it's achievable more often than people expect:
| Stage | Target |
|---|---|
| Local unit suite | under 3 minutes |
| Full unit suite in CI | under 5 minutes |
| PR pipeline (unit + lint + build) | under 10 minutes |
| Nightly full run with integration and e2e | under 45 minutes |
If you can't hit ten minutes, at minimum make sure a developer can answer 'did I break the thing I just touched' in under two.
The audit checklist
Work through these in order and you'll usually find the biggest wins in the first hour:
- Profile and record your 25 slowest tests with file names.
- Classify every test file as unit or integration, in writing.
- Count how many 'unit' tests touch the network, disk, or a database.
- Pull failure rates over the last 30 days and list every test with a retry-then-pass history.
- Check whether CI shards use timing data or naive splitting.
- Measure your dependency cache hit rate — anything below 80% is leaking minutes.
- Confirm test order is randomized with a fixed, logged seed.
- Replace every
sleep()with fake timers or polling assertions. - Kill any test that depends on a real third-party API.
- Measure p50 and p95 of the local test command, not just the average.
The test suite is a product too, and its users are your own engineers. Treat it with the same respect you'd give an endpoint that pays the bills — because every minute it wastes, it wastes on someone who was trying to do the right thing.