The Code We Haven't Read

Imagine a subscription service in the middle of an incident. A customer canceled, then reactivated before their data was due to be cleaned up. They can sign in again, but some of the data they expected to find is gone. The engineers have the account open alongside the incident channel. They don't yet know whether this is the only customer affected.

There's a cleanup job meant to remove data after cancellation. It's an obvious place to look, but nobody has established why this customer's data disappeared. The team needs to stop any further deletion while it investigates. It also needs to understand a piece of implementation nobody has visually reviewed.

That's what I mean here by dark code: agent-written code not visually reviewed by human eyes. It may have tests. It may have been running without a reported problem. The team accepted it into the system, and now the behavior it needs to explain is a customer's missing data.

The source is available. An engineer can open it, and an agent can help trace it. But availability isn't familiarity. During the incident, the team is trying to learn what the implementation does while also deciding what it can safely do next.

What did we think we knew?

Ask a coding agent for a whole feature, or a refactor that reaches across several components, and the implementation and tests can pile up together. By the time the work is ready for review, you may be looking at thousands of changed lines. It's still one feature. There's just a lot of it to read.

Line count doesn't tell us how difficult that work is to understand. A mechanical change can be large and straightforward; a small change can alter when we're allowed to delete customer data. But generation gives us a way to accumulate implementation faster than we accumulate familiarity with it. We can keep accepting work while becoming less able to explain how the pieces interact.

We haven't suddenly gone from complete understanding to none. Engineers have always operated systems they didn't write and couldn't hold entirely in their heads. Reading code doesn't guarantee understanding it, either. The question is what happens to our confidence when the work can arrive with an implementation, a test suite and an explanation, all produced without anyone having examined the details.

Those are useful things to have. They can also agree with one another for the wrong reason. If the requirement was misunderstood, a tidy implementation and a thorough-looking set of tests may faithfully preserve the misunderstanding.

Before the incident, the team may have understood the subscription contract perfectly well: reactivate before cleanup and keep your data. What it didn't establish was whether that contract survived the path through the running system. The cancellation request, the delayed job and the reactivation each look reasonable in isolation. The customer experiences them together.

A good explanation is a beginning

Suppose the agent traces the cleanup path and finds that the delayed job acts on the cancellation recorded when it was scheduled. It doesn't establish that deletion is still authorized when it runs. That would explain how a customer could reactivate and still lose data later.

That's a hypothesis the team can investigate. The account timeline, job executions, deletion records and relevant implementation can support or contradict it. If the engineers reproduce the failing transition in an isolated environment, they have something more useful than a plausible story. They've established a particular failure under particular conditions.

The agent is valuable here. It can follow references across components, compare events and help build the reproduction. Refusing its help because an agent wrote the original code would be an odd response to an incident. The concern isn't whether the explanation came from an agent. It's what supports it.

Even a verified explanation leaves work unresolved. Finding the path that deleted this customer's data doesn't establish every customer it affected. It doesn't tell us whether another delayed operation uses the same assumption. The investigation has made the system less mysterious, but only where it has reached.

That's an uncomfortable place to make decisions from. It is also a fairly ordinary place to be during an incident. Dark code makes me wonder how much of that investigation we're postponing until something goes wrong, and how well we'll recognize the gaps before then.

Test the outcome

One proposed answer is to test what matters to the customer rather than treating the shape of the implementation as the thing we need to approve. We don't necessarily have to read every line to establish that a meaningful outcome holds.

In this example, the outcome isn't that the cleanup job completes successfully. It isn't even that the reactivated account is marked active. The customer should still be able to access the data they were entitled to keep after the scheduled cleanup has had an opportunity to run.

A test that exercises that sequence could catch a failure that separate tests of cancellation, reactivation and cleanup miss. It gives us a way to challenge behavior across a boundary that matters, without requiring everyone to understand every function involved. That's a useful form of evidence, not a consolation prize for failing to read the source.

But the test is source too.

If an agent writes it and nobody visually reviews it, we have what I'll call a dark test. It might be an excellent test. It might also be named reactivated_customer_keeps_data while asserting only that the subscription is active. The name describes the outcome we wanted. The assertion establishes something smaller.

A green result doesn't show us that difference. Neither does a count of passing tests. We can have a great deal of verification machinery running and still misunderstand what it verifies.

The shared-assumption problem matters here. If the implementation treats cancellation as permanent authority to delete, tests built from that implementation might never challenge that authority after reactivation. Asking another agent to review them may help, but a separate conversation isn't, by itself, evidence of independent reasoning. Both may work from the same incomplete description.

There are ways to challenge the tests as well. We could deliberately introduce the deletion behavior we're worried about and see whether the outcome test fails. If it stays green, we've learned something concrete about its limits. If it fails, we've gained evidence that it detects that failure. We haven't established that it detects every consequential variation, or that we chose all the outcomes worth protecting.

I don't take that as a reason to dismiss generated tests. Unread tests can catch real defects, and a human can read an assertion and still miss what it leaves out. But testing for outcomes doesn't remove the question of understanding. It moves part of it into deciding which outcomes matter and establishing what the tests actually demonstrate.

Less uncertainty isn't certainty

There are other ways to make this situation less dangerous. Keeping the rules around destructive operations small enough to inspect could help the team understand why deletion is authorized. Restricting what a job can delete could limit the damage if that understanding is wrong. Neither requires us to pretend the entire system is understood.

Those approaches have limits too. A deletion rule may be short while depending on state maintained elsewhere. A narrowly scoped operation can still remove the one thing a customer needed. The useful discussion isn't whether these measures are perfect, but what confidence each gives us and what remains outside it.

Recovery belongs in that discussion. A patch that prevents another deletion doesn't bring back the missing data. A backup may offer a recovery path, but restoring it could overwrite legitimate changes made afterward. Until the team has established what it can recover and how, recoverability is another assumption, not an outcome.

The agent can help examine those assumptions, prepare recovery work and keep track of unanswered questions. That doesn't make it responsible for deciding how much uncertainty the team should accept on behalf of its customers. The team still owns that decision, even when the evidence takes more work to understand than the implementation took to generate.

After the explanation

Return to the engineers with the customer's account open. Suppose they've reproduced the deletion and shown that a proposed change prevents it under the conditions they tested. That's real progress. It doesn't settle whether the missing data can all be recovered, and it doesn't establish the behavior of every other path nobody has read.

We shouldn't dismiss what they've learned because it isn't complete. We also shouldn't let confidence in that one explanation quietly become confidence in the rest of the system.

Objective outcome testing gives us part of an answer. If we can show that a reactivated customer keeps their data after cleanup runs, and that the test catches a change that would delete it, we’ve established something useful without reading every line. The implementation isn’t fully understood, but that behavior is no longer just an assumption.

What remains harder to judge is how far that evidence reaches. Which important outcomes haven’t we tested? What conditions haven’t we considered? And how will we notice when our confidence in the system grows beyond the behavior we’ve actually demonstrated?