The Question That Became Clutta

In April 2019, five years before I began building Clutta, I put a sentence on slide sixteen of a research presentation:

Specify threshold for expected drop rate minus experienced drop rate.

The presentation was at Schloss Dagstuhl, during a working retreat on software-defined networks for critical infrastructure. The sentence sat under a heading called Latent Error Detection. It was one bullet among twenty-nine slides, describing work I thought belonged to smart-grid networks.

I did not know that I would keep meeting the same problem in different systems. At the time, it belonged to the network in front of me.

The question before the name

The slide was not the beginning. In 2011, Fault Tolerant Computing gave me the formal language for systems that continue while their parts fail. During my master's work, I built a telecommunications data layer across MySQL, MongoDB and Redis, then placed it inside a live mobile-core testbed. The evidence chapter included a 69-second phone call.

That call mattered because individual components could appear healthy without proving that the system they formed actually worked. Databases could contain the right rows. Interfaces could return successful responses. Processes could remain alive. I wanted the behaviour the system existed to support.

The instinct was simple: do not confuse signs of life with proof of work.

I carried it into Germany, where the system under investigation was no longer a mobile core but a critical network. I implemented IEEE 802.1CB frame replication in P4, initially working towards programmable network hardware and ultimately running the full experiment across a nine-switch software testbed.

The standard protects a stream by sending copies along separate paths. If one path fails, another copy can still arrive. The application keeps working. That is the success.

It is also the danger. A surviving copy can conceal the loss of the redundancy that made the system safe. Everything important still appears to work, but the system is now one failure away from having no answer left. The fault has not stopped the service, so the service does not announce the fault.

That was the latent error on slide sixteen: damage hidden inside apparent success.

A catalog that lied politely

Years later, the same failure shape appeared in a data catalog.

The catalog ingested metadata from Kafka and its schema registry. The ingestion completed successfully. The service stayed up. No alarm fired. The catalog simply attached the wrong schema to a topic whose name shared a prefix with another topic.

Nothing crashed. The catalog asserted a small falsehood confidently, and every engineer who trusted the catalog inherited it.

The correction landed upstream in DataHub as pull request 8224: two files, eighteen inserted lines. The size of the fix was not the lesson. The failure was. Availability and correctness had quietly separated, and the system's own signals had nothing to say about it.

The same issue appeared again in production automation.

I built a daemon that continuously found suspended subscriptions and restored the ones that qualified. It replaced a human procedure with software that ran continuously. The process running was not the proof I cared about. The customer reaching the correct state was.

By then I had met the same question in coursework, research, infrastructure and business systems.

Observation is testimony, not proof

Green can mean a process responded, a threshold was not crossed, a request succeeded, or a test passed. Each fact is useful, but each proves only what it measured.

This is why silent failures are so expensive. They do not defeat the monitoring system. They pass through it. The system reports exactly what it was asked to report, while the user experiences something the report was never designed to describe.

The lesson I took was not that health checks, metrics, logs or traces were useless. They are indispensable. The lesson was narrower and more demanding: evidence must never be allowed to claim more than it proves.

The record before Clutta

Origin stories are easy to retrofit. These dates come from files created before Clutta existed.

There is a transcript from 2011. A thesis with a 69-second call from 2016. A slide from April 2019 carrying the phrase latent error detection. In 2020, an idea diary records telemetry services for infrastructural problems. In 2021, the same diary records a tool to help support teams gather customer incidents reliably. In 2023, a public pull request corrects a catalog that was healthy and wrong. By 2024, a production daemon is acting on business state while its author is still preoccupied with how such action earns trust.

None of these artifacts is an early Clutta plan. Together, they show the same engineering instinct recurring: do not confuse activity with correctness, and do not let evidence claim more than it proves.

Starting Clutta

In 2024, I stopped treating the problem as a recurring feature request inside unrelated systems and began building Clutta.

Clutta's public description is compact: observability for failures that pass every health check. This essay is the history behind that sentence. The product must prove the rest.

The systems changed. The question did not. Clutta is where I chose to keep working on it.

Do not ask only whether the system is alive. Ask what its green light actually proves.

The company began in 2024. The question is much older.