AI-Assisted Testing — Doing It Right

The Problem with Letting AI Write Your Tests

There is a widely shared piece of advice in AI development communities: "Use AI to write your tests." The advice is right about the efficiency gains — AI can generate test scaffolding and common cases faster than you can type. But it is incomplete without an important caveat: when AI generates both the implementation and the tests for it, the tests are structurally likely to be wrong in a specific and dangerous way.

This failure mode has a name: tautological testing. A tautological test verifies what the code does rather than what it should do. It passes not because the code is correct, but because the test was written from the same set of assumptions as the code.

Here is a concrete example. Suppose a function should calculate sales tax at 8%. Due to a business rule you forgot to mention, the AI generates the calculation at 10%. The AI then generates tests for that function. The tests pass — not because the calculation is right, but because the tests assert exactly what the function does. You now have green tests on broken code, and the coverage report looks excellent.

Why This Happens

When AI generates both the code and the tests in the same session, the tests inevitably reflect the code's internal assumptions. The AI observed what the function does and wrote assertions that match it. There is no external reference point — the tests are circular.

This is not a subtle edge case. It is a well-documented failure mode that practitioners encounter regularly when working with AI-generated code. The research is unambiguous: the only reliable way to prevent tautological tests is to write test descriptions before writing the implementation.

The TDD Inversion

The solution is a simple discipline: separate the writing of test descriptions from the generation of code and test code, in that order.

The workflow looks like this:

• Step 1: Write test descriptions in plain English — what the function should do, what it should return for specific inputs, what edge cases it should handle. Do not write code yet.
• Step 2: Review the test descriptions against your requirements. This is the human verification step. Do the descriptions match what the feature actually needs to do? Add any missing edge cases you can think of.
• Step 3: Ask AI to generate test code from the confirmed descriptions. "Turn these test descriptions into [Jest / pytest / RSpec] test code. Do not write the implementation."
• Step 4: Ask AI to generate the implementation that makes those tests pass.

The critical difference: in Step 3, the AI is generating test code from your requirements (the descriptions), not from an implementation it just wrote. The tests are grounded in external requirements rather than circular code assumptions.

Where AI Testing Is Genuinely Useful

With the tautological test risk managed, AI is a real accelerant for several testing tasks:

• Edge case generation: Given a function signature and a list of known cases, AI is better than humans at generating an exhaustive list of boundary conditions and unusual inputs. Prompt: "What inputs could make this function fail silently or produce unexpected output? Consider: null/undefined, boundary values, empty collections, very large inputs, concurrent calls, unexpected types."
• Scaffolding boilerplate: Test setup, teardown, mock configuration, and fixture creation are time-consuming and largely mechanical. AI handles this well and faster than humans.
• Coverage expansion: Given a set of tests you have already written, AI can identify the paths not covered and suggest additional cases. This works because you wrote the originals and they are grounded in actual requirements.

Mutation Testing: The Verification Layer

Even with the TDD inversion in place, it is worth knowing about mutation testing as a way to verify that your tests actually catch bugs. Mutation testing tools (mutmut for Python, Stryker for JavaScript and TypeScript) automatically introduce small changes to your code — changing >= to >, removing a return statement, flipping a boolean — and check whether your tests detect these mutations. Tests that fail to catch mutations are tests that would not catch the corresponding real bugs.

Running mutation testing periodically on critical modules gives you an objective measure of test quality that coverage percentages cannot provide. A function with 80% test coverage that catches 30% of mutations has a much weaker test suite than it appears.

The Summary Rule

AI + tests = great for scaffolding, edge case generation, and coverage expansion. AI + tests = dangerous when AI writes both the implementation and the assertions in the same session. The discipline that prevents the problem is writing requirements-grounded test descriptions before any code generation begins. It takes an extra few minutes upfront and prevents the kind of false confidence that ships broken code behind green tests.