Automation testing is supposed to save time. But in practice, we regularly take over projects where the automated test suite is more of a liability than an asset — brittle tests, hour-long run times, zero developer trust.
This is more common than people admit. Here’s what actually goes wrong.
The five failure modes we see most often
1. Automating too early
The most common mistake. Teams reach for automation before the product has stabilised — automating UI flows that change every sprint. Every feature change breaks fifteen tests. Developers start seeing test failures as noise rather than signal.
Fix: Automate stable paths first. If a feature has changed three times in the last month, it’s not ready to automate.
2. Testing at the wrong layer
UI end-to-end tests are expensive to write and maintain. A lot of what teams write as E2E tests should be API tests, and a lot of what they write as API tests should be unit tests. The classic testing pyramid is ignored in favour of automating what’s visible.
Fix: Before automating anything, ask: what’s the lowest level I can test this at? UI tests should cover user journeys, not edge cases.
Caption goes here as italics — styled as a caption automatically
3. No ownership
Test code that lives in a shared folder with no clear owner gets stale fast. When tests fail, nobody knows if it’s a real bug or a flaky test. Nobody fixes them. The suite gradually becomes useless.
Fix: Treat test code like production code. It needs ownership, review, and maintenance. On our projects, the QA engineer owns the automation suite — not the dev team.
4. Ignoring test data
Automated tests need consistent, repeatable data. Teams often run tests against shared environments with inconsistent state, causing failures that can’t be reproduced. Faker.js exists for a reason.
Fix: Tests should generate or own their own data. Shared state between test runs is a recipe for intermittent failures.
Use playwright test --reporter=html to generate reports.
test('user can log in', async ({ page }) => {
await page.goto('/login');
await page.fill('#email', 'user@example.com');
});| Area | What AI catches well | What it misses |
|---|---|---|
| Correctness | Off-by-one errors | Business logic gaps |
| Readability | Long functions | Misleading names |
| Testing | Missing assertions | Test strategy flaws |
5. No integration with CI/CD
If tests only run when someone remembers to run them manually, they’re not actually automated. We’ve taken over projects where the “automation suite” runs once a week, by hand.
Fix: Tests that aren’t in the CI pipeline aren’t real automation. This should be day one infrastructure, not a nice-to-have.
What good looks like
The healthiest automation setups we’ve worked on share a few traits:
- Fast feedback loop — full regression under 20 minutes in CI
- Clear failure signal — when a test fails, it means something is wrong, not “flaky test again”
- Layered coverage — unit tests at the bottom, API tests in the middle, a smaller number of E2E tests covering critical user paths at the top
- Maintained like code — PRs, reviews, refactoring when the product changes
Getting there usually requires a QA audit first — understanding what’s actually being tested, what the coverage gaps are, and what needs to be rebuilt versus salvaged.
If your test suite is causing more pain than it prevents, that’s worth fixing. It doesn’t require starting over — it requires a clear plan.