The Question That Became Clutta
A slide from 2019, a catalog that lied politely, and one engineering question that kept surviving every change of system.

Observability has become excellent at telling us what happened. It is still remarkably poor at telling us what should have happened. Most of what I write is about closing that gap.
A slide from 2019, a catalog that lied politely, and one engineering question that kept surviving every change of system.
A closed incident proves that a workflow ended. It does not prove that the customer recovered, the system stayed correct, or the same failure will not return under a new number.
A maintenance window is a confession: somewhere, an operation was never made bounded, observable, or abortable. What rebalancing 50,000 Kafka replicas taught me about removing the confession.
Availability, designed from a place where the power grid is part of your threat model. Reliability lessons the textbooks cannot teach.
Publishing in order as each is finished. Subscribe and they arrive the day they land.
Beyond the Three Pillars: Correlating Signals at Scale
Metrics, logs and traces can reconstruct what happened. There is an information bound on what they can ever tell you about what should have.
When Health Checks Lie: Anatomy of a Cascading Failure
A 200 proves a process is alive. It proves nothing about whether the work it is doing is correct, and that gap is where cascades are born.
Real-time observability for silent failures. Built on the premise that detection is a verification problem, not a collection problem.
Go · Kafka · OpenTelemetry · WebSocket · early access
A rebalancer for production Kafka, built solo: 17,000+ partition leaders and 50,000+ replicas redistributed in batches of 200, gated by broker disk thresholds. Rebalancing stopped being a maintenance window and became a command.
Python · Kafka internals · Salt · in production · field notes ↗
Designed and implemented end to end: a Go daemon for a service-continuity platform that continuously discovers suspended subscriptions and restores the ones that qualify. Remediation as a daemon rather than a runbook.
Go · Redis · GraphQL · Nomad · in production
First a topic-ownership service, designed and implemented end to end, aggregating ACLs, log dirs and consumer groups. Then, having hit the limits of hand-rolled metadata, DataHub: ElasticSearch reliability and schema-registry integration, contributed upstream to LinkedIn's open source.
Python · Java · ElasticSearch · Confluent Schema Registry
RCS is now widely supported across Android and, since 2024, iPhone: a decade after this work, it has become a mainstream mobile messaging standard, with end-to-end encryption arriving in the Universal Profile. A decade before that, I built the subscriber-data layer such networks need: MySQL, MongoDB and Redis exposed as one OData repository inside a live IMS core. Infrastructure for a standard the world had not adopted yet.
Java EE · OData · MySQL · MongoDB · Redis · Open IMS Core · 11,575 LOC · read the thesis ↗
First P4 implementation of IEEE 802.1CB frame replication, built clause by clause on a nine-switch bmv2 smart grid testbed, with the ONOS P4Runtime driver written to support it. Worked the hardware path on Netronome SmartNICs before running the full testbed on software switches: same proof, none of the cost.
P4 · ONOS · bmv2 · Mininet · SDN · NFV · live testbed
From deep systems mastery to shaping resilient infrastructure, building teams, and crafting tools for others to scale.
Diagnosing, dissecting, and restoring at enterprise scale. The years that proved dashboards can be green while the system lies.
Research conducted the way production is run: live testbeds, working code, and specifications implemented rather than cited.
Delivering working systems without external scaffolding. No frameworks, no teams, often no prior blueprint. Every line of code had to matter.
If you are deep in a system that is failing in ways your dashboards cannot explain, that is the conversation I most want to have. I read every email.