TDD with AI coding agents: keep the tests honest

An AI coding agent can make a bad test pass very quickly.

That is the central risk in AI-assisted TDD. The agent can write the test, production code, mocks, and fixtures in one context. If the requirement is vague, both sides may share the same mistaken assumption. Green proves consistency between two generated artifacts, not correctness.

The useful workflow keeps the feedback loop small and makes every transition observable.

TDD is one behavior at a time

The familiar cycle is red, green, refactor:

1. write the next test for one behavior;
2. run it and see the expected failure;
3. write enough production code to pass;
4. run the relevant suite;
5. improve structure without changing behavior.

Martin Fowler’s TDD summary also starts with a test-case list and then selects one test for each cycle. The sequence of tests helps drive the design (Fowler). Asking an agent to generate twenty failing tests and then implement everything is test-first batch development, not the tight TDD loop.

AI is useful because it can read local conventions, edit quickly, run commands, and respond to failures. Anthropic’s current Claude Code guidance emphasizes specific context and a verification target; its agentic loop explicitly uses tests to check work (best practices, how Claude Code works).

The agent still needs an oracle: a human-approved contract, example, protocol, or existing behavior.

Step 0: make the environment deterministic

Before starting a red-green cycle, write down:

Target test command:
Full verification command:
Files allowed to change:
Files that must not change:
External services:
Clock/randomness policy:
Fixture and cleanup policy:

Run the existing target suite. If it is already red or flaky, record that separately. Otherwise the agent may attribute an old failure to its change or “fix” unrelated tests.

Pin controllable sources of nondeterminism:

• inject or fake time rather than sleeping;
• seed randomness or isolate random policy;
• use owned fixtures;
• avoid a live third-party API in the inner loop;
• reset database state;
• make locale and timezone explicit.

Do not add a testing dependency merely because the agent knows it. Follow the repository’s existing framework and approval process.

Step 1: write a behavior contract

Tests are executable examples, not the entire specification. Start with observable rules and exclusions.

Example for a retry-delay function:

Function:
calculateRetryDelay(attempt, baseDelayMs, maxDelayMs) -> number

Rules:
- attempt is a zero-based non-negative integer
- delay doubles for each subsequent attempt
- result never exceeds maxDelayMs
- baseDelayMs and maxDelayMs are positive integers
- maxDelayMs must be at least baseDelayMs
- invalid input throws RangeError
- no jitter in this function

Non-goals:
- deciding whether an operation is retryable
- sleeping
- logging
- reading environment variables

A useful prompt points to the repository rather than restating it from memory:

Read the existing tests and implementation conventions in:
- src/retry/
- test/retry/

From the approved contract below, propose an ordered test list.
Do not edit files and do not design extra behavior.
For each test, state the observable rule it proves and why it is next.

[contract]

Review the list. Remove duplicate examples and invented requirements. Choose the smallest test that moves the design.

Step 2: create exactly one red test

Prompt:

Add only the first test from the approved list.

Constraints:
- modify the designated test file only;
- do not create or edit production code;
- follow existing Vitest conventions;
- assert public behavior, not private implementation;
- run: npm test -- retry-delay.test.ts;
- report the exact failing assertion or error.

Stop after proving red.

A first test might be:

import { describe, expect, it } from 'vitest';

import { calculateRetryDelay } from './retry-delay';

describe('calculateRetryDelay', () => {
  it('returns the base delay for attempt zero', () => {
    expect(calculateRetryDelay(0, 100, 1_000)).toBe(100);
  });
});

Inspect three things:

1. the test fails before the production change;
2. it fails because the behavior is missing;
3. the failure would disappear only if observable behavior changes.

“Module not found” can be acceptable for the very first slice, but a syntax error, broken fixture, or missing test dependency does not prove the product behavior. If the test passes immediately, investigate. Do not pretend the red step happened.

For a bug fix, reproduce the user-visible defect:

Add one regression test that fails on the current branch and represents:
[observed input, state, and expected output].

Do not edit production code.
Do not generalize beyond the incident yet.
Run the narrow test and show the failure.

Step 3: make the smallest green change

After approving red:

Make the smallest production change that passes the new test.

Constraints:
- do not modify, delete, skip, or weaken tests;
- do not add unrequested behavior;
- keep public API changes inside the approved contract;
- run the narrow test, then the existing retry suite;
- stop and explain if the test itself appears wrong.

The minimal implementation can be deliberately small:

export function calculateRetryDelay(
  attempt: number,
  baseDelayMs: number,
  maxDelayMs: number,
): number {
  return baseDelayMs;
}

That is enough for the first behavior. It is not the final algorithm.

Review the diff, not only the exit code. Agents sometimes make green by changing a fixture, broadening a matcher, replacing a real dependency with a mock, or catching the error under test. Keep test changes out of the green commit unless a human explicitly approves a correction.

Step 4: continue with boundary cases

Add one behavior per cycle:

it('doubles the delay for each attempt', () => {
  expect(calculateRetryDelay(1, 100, 1_000)).toBe(200);
  expect(calculateRetryDelay(2, 100, 1_000)).toBe(400);
});

it('caps the delay at the configured maximum', () => {
  expect(calculateRetryDelay(5, 100, 1_000)).toBe(1_000);
});

it.each([
  [-1, 100, 1_000],
  [0.5, 100, 1_000],
  [0, 0, 1_000],
  [0, 100, 99],
])('rejects invalid inputs', (attempt, baseDelayMs, maxDelayMs) => {
  expect(() =>
    calculateRetryDelay(attempt, baseDelayMs, maxDelayMs),
  ).toThrow(RangeError);
});

Before accepting an agent-suggested edge case, ask which contract rule or failure mode justifies it. null, empty arrays, timeouts, malformed UTF-8, and concurrency are not universal decorations. They belong only when the type boundary, runtime, or domain makes them possible.

The final implementation may be:

export function calculateRetryDelay(
  attempt: number,
  baseDelayMs: number,
  maxDelayMs: number,
): number {
  const hasInvalidInput =
    !Number.isInteger(attempt) ||
    attempt < 0 ||
    !Number.isInteger(baseDelayMs) ||
    baseDelayMs <= 0 ||
    !Number.isInteger(maxDelayMs) ||
    maxDelayMs < baseDelayMs;

  if (hasInvalidInput) {
    throw new RangeError('Invalid retry delay parameters');
  }

  return Math.min(baseDelayMs * 2 ** attempt, maxDelayMs);
}

For very large attempt, this expression can overflow to Infinity; Math.min still returns the cap for finite valid configuration. Whether to reject unsafe integers is another contract decision, not something the agent should silently choose.

Step 5: refactor while green

Refactoring changes structure without changing observable behavior. Give the agent a bounded goal:

The retry tests are green.

Refactor only src/retry/retry-delay.ts to make validation easier to read.
Do not change public behavior or tests.
Run the narrow test after the edit and the full test command afterward.
Show the diff and both command results.

A refactor step is optional. Do not add abstraction because the cycle has a box named “Refactor.” Remove duplication or clarify a design pressure that exists now.

Run format, type, lint, build, and relevant integration checks defined by the repository. A unit suite does not prove the package compiles or the exported API remains compatible.

Keep the test oracle independent

AI-assisted TDD becomes circular when the same vague prompt produces the requirement, test, and implementation.

Use at least one independent source:

• a protocol or public API contract;
• an accepted bug report with reproduction;
• a domain example approved by a subject-matter expert;
• a database constraint;
• a recorded request and response fixture;
• a reference implementation;
• a manually calculated example.

Then separate roles in the workflow:

Contract review:
Find ambiguity and missing decisions. Do not write code.

Test author:
Use the approved contract and existing test style. Do not inspect or edit
the planned implementation.

Implementer:
Use the approved test. Do not change it.

Challenger:
Inspect contract, tests, and implementation. Propose a minimal counterexample
that could pass the suite while violating the contract.

These roles can be separate sessions or explicit phases. Separate sessions reduce shared assumptions, but they do not create independent ground truth. The contract remains the source.

For sensitive calculations, manually verify a few examples. For parsers, keep accepted and rejected fixtures. For state machines, enumerate transitions. For security boundaries, pair functional tests with the AI code-review checklist and threat analysis.

Measure test strength, not just coverage

Line and branch coverage answer which code executed. They do not prove that assertions would catch a wrong result.

Challenge important tests in increasing order of cost:

1. Known-bad edit: temporarily reverse a comparison or remove validation. The test should fail.
2. Boundary table: values immediately below, at, and above each boundary.
3. Invariant: output stays within the declared range; an operation remains idempotent where required.
4. Mutation testing: automatically inject small faults and report which survive.
5. Real defect replay: keep a regression test for each relevant production bug.

Research on LLM test generation reinforces the distinction: a 2024 study used mutation feedback to improve fault-revealing ability instead of treating coverage as sufficient (Dakhel et al.). That result does not mean an arbitrary generated suite achieves the paper’s score or transfers to every language and codebase.

If a mutation tool is already approved, run it on the changed module rather than the entire repository first. Surviving mutants are review prompts, not an automatic demand for more tests; equivalent mutants and irrelevant implementation details exist.

Do not over-mock the path you need to prove

A test can be fast and useless:

billingClient.charge.mockResolvedValue({ status: 'paid' });
expect(billingClient.charge).toHaveBeenCalled();

This proves that the mock returned what the test configured. It does not prove the webhook signature, idempotency key, database transaction, or provider mapping.

Place evidence at the boundary of the risk:

For AI systems themselves, use a separate agent evaluation plan. A probabilistic LLM judge inside a unit test needs versioning, thresholds, cost controls, and calibration; it is not a drop-in Boolean oracle.

Put the rule in the repository

Prompts disappear. Encode the workflow in project guidance or a reusable command:

For behavior changes:
1. confirm the contract and test command;
2. add one failing test;
3. run it and report the expected failure;
4. do not edit production code until red is approved;
5. make the smallest production change;
6. never change a test to force green;
7. run narrow and full verification;
8. review the diff before commit.

CI should rerun the authoritative checks in a clean environment. The local agent transcript is useful evidence, not the release gate. Connect the suite to the existing CI pipeline with the same commands developers run locally.

When AI-assisted TDD is a poor fit

Do not force this loop when:

• the team is exploring an unknown interaction and cannot yet define behavior;
• the observable outcome is primarily visual and no stable reference exists;
• a one-off migration is better verified through reconciliation;
• legacy code has no seam for a meaningful test without a larger characterization effort;
• the required oracle is a domain decision nobody has made.

Run a spike, collect examples, or write characterization tests first. Delete throwaway code or keep it clearly separate from production work. TDD works when the next behavior can be stated and observed; AI does not remove that requirement.

Compact Claude Code prompt sequence

1. Read [contract] and [existing test examples].
   Propose an ordered test list. Do not edit files.

2. Add exactly the next test in [file].
   Do not edit production code.
   Run [narrow command] and prove the expected red.

3. Make the smallest production change.
   Do not modify tests.
   Run [narrow command], then [full command].

4. Review the diff against the contract.
   Find any way the implementation can violate the contract
   while all current tests pass.

5. If structure now needs improvement, refactor while green.
   Run format, types, tests, and build.

The strongest part of TDD with a coding agent is not faster test generation. It is the visible feedback loop: one approved behavior, one meaningful red, one bounded green change, and independent evidence that the test can catch a fault.

Sources

• Martin Fowler: Test Driven Development
• UK Government Digital Service: TDD standard
• Claude Code: best practices
• Claude Code: common testing workflows
• Claude Code: how the agentic loop verifies work
• Dakhel et al.: LLM test generation with mutation feedback