Replay the Clock, Not Just the Data

A backfill that rewrites time is not a backfill. It is a new event with an old payload.

I learned this properly while trying to establish a baseline. I had roughly two thousand historical records that needed to exist in a system so that it could learn what normal looked like before I asked it to judge anything. So I wrote a replay tool, pointed it at the archive, and watched it succeed. Two thousand records in, zero errors, exit code zero.

Then I looked at what the system had learned, and it had learned nothing, because as far as it was concerned every one of those two thousand events had happened in the same instant.

What the tool did

The replay loop built each outbound event and, for the event time, called the equivalent of time.Now(). Not the timestamp on the archived record. The clock on the wall at the moment of sending.

The payloads were faithful. Identifiers, durations, statuses, every field carried across correctly. Only the one field that located the record in history was overwritten, and it was overwritten with something that looked entirely reasonable: a valid, well-formed, recent timestamp.

Here is the arithmetic that makes it a real problem rather than an aesthetic one. Spread across sixty days, two thousand records give you a distribution: a mean, a variance, a visible weekday rhythm, a sense of what a slow Tuesday looks like against a busy Friday. Collapsed into a single minute, the same two thousand records give you one data point with a count of two thousand next to it. Variance across time is exactly zero, because there is no longer any time to vary across.

Every subsequent judgement the system made was therefore made against a baseline that described a world where all activity for two months occurred in one minute. It did not refuse to make those judgements. Nothing in it could tell that the baseline was wrong.

Why nothing complained

This is the part worth sitting with, because it is the same shape as every other failure I have written about.

The replay tool had no bug in the ordinary sense. It did precisely what its code said. The target system had no bug either; it accepted a well-formed event with a valid timestamp, which is its job. The transport was fine. The schema validated. Both ends were correct, and the composition was wrong.

And there was no signal available that could have caught it. A count check passes: two thousand sent, two thousand received. A schema check passes: every field present and correctly typed. A spot check on a single record passes, because a single record has no timeline to be wrong about. The error only becomes visible when you ask a question about distribution, and nobody asks that question until they are already suspicious.

I found it because the output looked too clean. That is not a detection strategy.

The general rule

State it narrowly and it is obvious. State it generally and it is a trap most systems I have worked on have fallen into at least once.

Any system that replays, imports, migrates or backfills historical events must treat the original event time as part of the payload, not as metadata the transport is free to supply.

The failure has a family. Log shippers that stamp ingestion time and let you believe it is event time. Message queues where the only available timestamp is enqueue time. Metrics pipelines that bucket by scrape rather than by occurrence. Database imports where created_at has a DEFAULT now() and the import does not override it. In every one, the data survives and the chronology does not, and in every one the system reports complete success.

Three practices have saved me since.

Make event time a required input, never a default. If the function that emits an event can get a timestamp without being given one, someone will eventually let it. Remove the convenience and the bug class goes with it.

Assert on the span, not just the count. After any backfill, check the minimum and maximum event time against what you expected to load, and check that the spread is non-zero. That single assertion would have caught this in the first run. It costs one query.

Keep both clocks. Record event time and ingestion time as separate fields, always. When they disagree you learn something real about your pipeline, and when one is missing you at least know which one you are looking at.

None of this is sophisticated. The reason it is worth writing down is that the system on the receiving end was itself a monitor, one that learns a baseline before it is allowed to alert, and the replay fed it two thousand events with a fabricated history and called the run a success. The tools that watch your systems are systems. They fail the same way, and they are watched by nobody.