Three Regression Tests for AI-Assisted Code

For AI-assisted changes, begin with the smallest test that reproduces the failure—not a broad test the new code is likely to pass. Recursive callbacks, expired identity, and idle connections show how narrow regression tests constrain an agent usefully.

The examples below are from public repositories I maintain: GitLab MCP, MediaWiki MCP, and Buddy Bridge. None of the bugs was caused merely by using an agent. They are ordinary software bugs. An agent just makes a vague stopping condition more expensive because it can keep producing plausible changes long after the important behavior is still wrong.

The test starts with the sentence I wish I had written earlier

“Closing an already-closing session must do nothing.” “A configured password does not prove the current session is authenticated.” “An empty queue can mean the connection is healthy and idle.” Those are better prompts than “add more tests” because they name the truth the program got wrong.

I try to reduce a production story to four parts:

1. The event: the exact input or lifecycle transition that starts the failure.
2. The observation: the response, state change, or side effect a caller can see.
3. The forbidden result: recursion, anonymous mutation, premature disconnect, duplicate work, or another outcome that must not occur.
4. The stopping command: one repeatable command that fails before the repair and passes after it.

That is the part of TDD for me. The test is not documentation about my intention. It is a small machine that can disagree with the implementation.

Test one: make the recursive callback call itself

My GitLab MCP server kept sessions in a map. Its close helper looked up a session, closed the MCP server, and then deleted the map entry. The order seemed harmless until the SDK lifecycle joined the call stack: closing the server closed its transport, and the transport’s onclose callback called the same helper again. The session was still in the map, so the helper found it and repeated the cycle until Node reported RangeError: Maximum call stack size exceeded.

A weak test would call the close helper once and confirm that the map eventually became empty. That misses the bug because a polite fake never re-enters the code. The regression test uses a deliberately impolite fake: its close() method synchronously invokes the close helper again.

The repair in GitLab MCP 2.2.1 deletes the session from the map before closing the server. On the nested call, there is nothing left to close. The test does not care how many helper functions exist or which line contains the deletion. It cares that teardown becomes true before teardown can announce itself again.

This is the kind of test I want an agent to run after every related edit. It is tiny, fast, and shaped like the real failure rather than the current implementation.

Test two: expire the identity but leave reads working

MediaWiki MCP had a more convincing lie. A bot username and password were configured, and the process remembered that it had logged in. Later, the MediaWiki session expired. Public searches and page reads continued to work, so the connector still looked healthy. A protected write no longer had the identity the local process claimed it had.

The important test does not simply mock a generic 401. It lets the token request succeed from a stale anonymous session, makes the protected action fail MediaWiki’s assert=user check, and then verifies a bounded recovery sequence: log in again, obtain a new CSRF token, retry once, and surface the error if the retry also fails.

The authentication repair added tests for bot-password reauthentication, anonymous clients, multipart uploads, and a failed retry. A separate status test proves that the tool reports the session MediaWiki can actually see, not the credentials sitting in configuration.

This changed how I write health tests. I no longer treat one successful public read as evidence that authenticated writes are healthy. The test must cross the boundary that owns the claim. MediaWiki, not an in-memory boolean, decides whether the request has an authenticated user.

Test three: make nothing happen

Buddy Bridge sends session state from several computers to a small ESP32-based display over an HTTP stream and Bluetooth. Its hub deduplicates identical heartbeats, which is sensible: if the state has not changed, there is no reason to keep sending the same payload.

The stream handler waited on a queue. When the queue stayed empty for the keepalive interval, Python raised queue.Empty. The handler treated that expected idle timeout as the end of the response. The relay reopened HTTP, and the first version also tore down Bluetooth. A system doing nothing correctly disconnected about once a minute.

Most connection tests make activity happen: connect, send, receive, close. The useful regression test makes the queue stay empty and checks that the stream emits a current heartbeat instead of ending. Companion tests verify that changed states still arrive, duplicate states remain suppressed until the keepalive floor, reader errors produce a teardown sentinel, and an HTTP reconnect does not automatically destroy the Bluetooth connection.

The current hub keepalive repair gives silence its actual meaning. This test is my favorite of the three because the trigger is the absence of an event. Without a clock and an explicit idle case, the failure is almost invisible in a normal happy-path suite.

What I hand to an agent

I get better results when the task includes a failure contract instead of a broad request to improve coverage. My working brief usually contains:

• the failing behavior in one sentence;
• the smallest files and subsystem in scope;
• the focused test command and the full-suite command;
• state or permissions that must not change;
• the expected failure before implementation, when a red test can be demonstrated safely;
• the evidence required at the end: test output, reviewed diff, and any limitation that was not exercised.

I do not ask the agent to invent the acceptance criteria after reading the implementation. That tends to produce a test that admires whatever code already exists. I describe the observable boundary first, then let the implementation move.

Fast feedback first, broader evidence second

A full suite is not a substitute for the focused failure. A focused failure is not a substitute for the full suite. One gives a sharp stopping signal; the other catches the assumptions I did not realize were shared.

Green is evidence, not a verdict

For the detailed reviews behind these examples, I ran 10 targeted GitLab session tests, all 201 MediaWiki MCP tests across 24 files plus its type check and production build, and all 59 Buddy Bridge tests. Those runs prove the checked code paths behaved as described in the reviewed checkouts.

They do not prove that I reproduced the original production traffic, waited through a real expired MediaWiki bot session, exercised every proxy, or paired and idled the physical display again. A fake callback can preserve a recursion fix. It cannot prove every SDK version will close in the same order. A fake clock can prove idle-expiry logic. It cannot turn a unit test into a load test.

I want that limitation in the article and in the agent’s handoff. Otherwise “all tests pass” becomes the next comforting proxy.

The review question that catches decorative tests

After the build goes green, I ask: what small mutation would make this test fail?

If moving the map deletion back below close() does not fail the recursion test, the fake is too polite. If removing the live user check does not fail the authentication test, the test is reading configuration instead of identity. If ending the stream on queue.Empty does not fail the idle test, the clock is not part of the fixture.

That question is more useful than a coverage percentage. Coverage can tell me a line ran. It cannot tell me the assertion would notice the lie that mattered.

What “TDD by design” means to me now

I kept the old URL because it had already received Google referrals, but I replaced the generic essay that used to live here. My rule is narrower and more practical now:

When a change will be made in an automated loop, the loop needs an executable disagreement. A test should be able to say that the session still exists too long, the identity is not real, or healthy silence was mistaken for failure. The agent can iterate on the code. I still decide which disagreement matters.

The full production stories are in the GitLab session failure, the MediaWiki false-health repair, and the Buddy Bridge idle-stream repair.

Disclosure: I maintain all three repositories used in these examples.