Marked Resolved, Still Failing

A ticket closing is an assertion about the world made by someone who is no longer looking at it.

I have been keeping a deliberately honest record of a long-running engineering effort, partly because I wanted the writing to be checkable and partly because I wanted to know whether I could trust my own notes. The answer, repeatedly, has been no, and the ways in which they failed turned out to be more interesting than the systems they were describing.

The tracker said closed

At one point I sat down to write a readiness assessment: what works, what does not, what blocks the next milestone. The responsible thing seemed to be to start from the register of known issues rather than from memory.

The register listed several issues as resolved that were not resolved. It listed others as open that had been fixed weeks earlier. The register was not careless; it had been maintained by someone paying attention, namely me. It had simply drifted, the way any cache drifts when there is no invalidation.

Then I found the better version of the same problem. I had been reading from a clone of the repository that was 163 commits behind the branch where the work had actually landed. So the assessment I was writing was not merely working from a stale summary of reality. It was working from a stale summary of a stale copy of reality, and at no point did anything tell me so. git log worked. The files were all there. The tests I ran passed. A repository that is a hundred and sixty-three commits behind looks exactly like a repository that is current.

The document I eventually published had to include a section correcting the document itself, because I found the drift while writing it and the honest move was to leave the correction in rather than silently fix the earlier paragraphs. I have come to think that is a feature. A status document that cannot record its own corrections is not a status document, it is a press release.

The counting problem

The same week, a smaller instance of the same shape, and this one I find genuinely funny.

A component had identified a set of candidate records locally. The count was 507. Those candidates were then synchronised to the store that everything downstream read from, and the count there was 407.

A hundred records did not make the trip. Nothing errored. The sync reported success. Neither number was flagged as surprising, because neither side knew what the other side's number was. The local side knew it had found 507 and had no opinion about what happened next. The remote side knew it held 407 and had no idea that was not all of them.

I want to be precise about why this stung. The component in question was a detector, built to find records that needed attention and hand them on. It had a silent incompletion inside itself. A detector is subject to the same failure modes it was built to detect, and it will not catch them in itself, because it is not pointed at itself.

The fix is embarrassingly small, and it is the only prescription in this essay worth copying down. Any time a count crosses a boundary, carry the count across with it and compare on the other side. Not a checksum, not a reconciliation job, just the number travelling alongside the thing it describes. Five hundred and seven went in, four hundred and seven arrived, and that comparison takes one line and would have turned a silent loss into a loud one.

The comment that named the bug it failed to fix

One more, because the pattern completes here.

I went looking for the history of a regression and found, in the issue thread, a comment I had written describing the problem correctly. The diagnosis was right. The mechanism was right. The proposed fix was right. And the regression was still live when I went looking, because the comment was the whole of the work. Writing the correct diagnosis had produced the full feeling of having addressed it.

That is a failure mode in a person rather than in a system, and it has the same signature as all the others in this essay. Every component reported success. The issue was triaged, understood, documented and discussed. The composition produced no change in the world, and nothing anywhere said so, because no instrument we use measures the gap between understood and fixed.

What I do differently now

I do not have a methodology to offer here, and I am sceptical of anyone who claims one for this. What I have is three habits that have each caught something real.

Treat the record as a replica, not as the source. Before planning from a tracker, a register, a status doc or a clone, check it against the thing it describes. For a repository that is one command. For an issue register it means opening the three issues that matter most and reading them rather than reading their titles.

Let documents correct themselves in public. When I find that something I wrote earlier in a document is wrong, I now add the correction rather than editing the original away. The correction is usually more informative than the content.

Count on both sides of every boundary. The cheapest reliability work available, and almost nobody does it, because a success that reports success feels like it has already been verified.

The common thread, and the reason I keep writing about this rather than moving on to more interesting problems: the systems we build to tell us the truth are systems, and they fail quietly, and the quieter they fail the more we trust them.