November 20, 2025
Here is a pattern I see constantly: an AI coding assistant generates a feature, writes 47 unit tests for it, then spends the next hour in a death spiral trying to make them all pass. The tests fail. The AI "fixes" them by changing the code. Now different tests fail. The AI changes the code again. Repeat until the original feature is unrecognizable.
Unit tests were supposed to give us confidence. Instead, with AI in the loop, they have become a liability.
The Problem: AI Treats Tests as Ground Truth
When a human developer sees a failing test, they ask: "Is the code wrong, or is the test wrong?"
AI assistants do not ask this question. They see a failing test and immediately start modifying code to make it pass. The test is treated as an immutable specification, even when the test itself is the problem.
This creates a perverse dynamic. The AI wrote both the code and the tests. When they conflict, the AI defaults to trusting its own tests over its own implementation. It will happily break working functionality to satisfy a test that was wrong from the start.
I have watched AI assistants:
- Change a function's return type to match a test's incorrect expectation
- Remove error handling because a test did not account for it
- Refactor working code into broken code because the test mocked something incorrectly
The tests become a straitjacket, not a safety net.
The Quantity Problem
AI is very good at generating lots of things. Ask it to write tests and you will get tests. Dozens of them. Hundreds if you let it.
Most of these tests are what I call "implementation mirrors." They do not test behavior. They test that the code does exactly what the code does. If you change the implementation, even to something functionally equivalent, the tests break.
Here is what happens:
- AI writes a function with specific internal logic
- AI writes 15 tests that verify that specific internal logic
- You refactor the function (same inputs, same outputs, different internals)
- All 15 tests fail
- AI "fixes" the function by reverting your refactor
The tests are not testing the contract. They are testing the implementation. And AI is particularly bad at knowing the difference.
The Wild Goose Chase
When tests fail in bulk, AI enters what I call "wild goose chase mode." It starts making changes to fix one test, which breaks another, which triggers a fix that breaks three more. The commit history becomes incomprehensible. The code becomes a patchwork of fixes-for-fixes.
I have seen AI assistants:
- Spend 45 minutes on a test suite that should have been deleted
- Create increasingly elaborate mocks to work around tests that tested the wrong thing
- Introduce actual bugs while trying to satisfy tests that were testing nothing useful
The tragedy is that the AI is being diligent. It is trying to make all the tests pass. That is exactly what we told it to do. The problem is that "make all tests pass" is not the same as "make the code correct."
What Actually Works: Integration Tests
Apple calls them UI tests. Others call them integration tests or end-to-end tests. The name does not matter. What matters is the approach: test the system from the outside.
Integration tests have properties that make them AI-resistant:
They test behavior, not implementation. An integration test says "when I click this button, this thing should happen." It does not care how the button click is processed internally. Refactor all you want. The test still passes if the behavior is correct.
They are harder to game. An AI cannot satisfy an integration test by tweaking internal assertions. The test exercises real code paths through real interfaces. Either the feature works or it does not.
They fail for real reasons. When an integration test fails, something is actually broken. When a unit test fails, maybe something is broken, or maybe the test was just too tightly coupled to implementation details.
They are fewer in number. You cannot generate 200 integration tests as easily as 200 unit tests. The friction is a feature. It forces focus on what actually matters.
The Testing Pyramid Needs to Flip
The traditional testing pyramid says: lots of unit tests at the bottom, fewer integration tests in the middle, even fewer end-to-end tests at the top.
This made sense when humans wrote all the tests. Unit tests were cheap to write and fast to run. Integration tests were expensive and slow.
But AI changes the economics. Unit tests are now free to generate but expensive to maintain (because AI breaks them constantly). Integration tests are still somewhat expensive to write but cheap to maintain (because they test stable interfaces).
Maybe the pyramid should flip. A few critical unit tests for genuinely complex logic. Many integration tests for actual user-facing behavior. Let the AI generate whatever tests it wants, but do not let those tests become the source of truth.
Practical Advice
If you are working with AI coding assistants:
Delete tests aggressively. If a test breaks every time you touch the code, it is testing implementation, not behavior. Delete it. The AI will not miss it.
Prefer integration tests. When you ask AI to write tests, ask for integration tests specifically. You will get fewer tests, but they will be more useful.
Question failing tests. When AI wants to change code to fix a test, ask: "Is the test correct?" Half the time, the answer is no.
Keep test suites small. A test suite of 500 unit tests is not a safety net. It is a minefield. Every change triggers a cascade of failures, and AI will happily spend hours navigating that minefield in the wrong direction.
Separate test runs. Run integration tests to verify behavior. Run unit tests only when you specifically want to verify internal logic. Do not let AI see both at once, or it will try to satisfy both simultaneously.
The Uncomfortable Truth
Unit tests were designed for a world where humans wrote code and needed mechanical verification. AI assistants do not need that verification. They can hold the entire codebase in context. They can reason about code paths directly.
What AI needs is behavioral guardrails. Tests that say "the system should do X" rather than "function Y should return Z." Integration tests provide that. Unit tests, as commonly written, do not.
I am not saying unit tests are useless. For genuinely complex algorithms, for tricky edge cases, for code that must be correct for safety reasons, unit tests still make sense. But the default mode of "generate unit tests for everything" is actively harmful when AI is in the loop.
The tests are supposed to serve the code. When the code starts serving the tests, something has gone wrong. And with AI assistants, that inversion happens constantly.
Maybe it is time to rethink what we are testing and why.
Bless up! 🙏✨