Silent Defaulting
A system cannot tell nobody told me from they told me a legitimate value. So it picks a default, carries on, and produces a wrong answer with no error anywhere.

Observability has become excellent at telling us what happened. It is still remarkably poor at telling us what should have happened. Most of what I write is about closing that gap.
A system cannot tell nobody told me from they told me a legitimate value. So it picks a default, carries on, and produces a wrong answer with no error anywhere.
I replayed two months of history into a system and every event arrived stamped with the moment of the replay. The data was perfect. The time was fiction. Nothing complained.
Your issue tracker is a distributed system with one replica per engineer and no consistency guarantee. It drifts from reality quietly, and then you plan against it.
A slide from 2019, a catalog that lied politely, and one engineering question that kept surviving every change of system.
A maintenance window is a confession: somewhere, an operation was never made bounded, observable, or abortable. What rebalancing 50,000 Kafka replicas taught me about removing the confession.
Availability, designed from a place where the power grid is part of your threat model. Reliability lessons the textbooks cannot teach.
Publishing in order as each is finished. Subscribe and they arrive the day they land.
When Health Checks Lie: Anatomy of a Cascading Failure
A 200 proves a process is alive. It proves nothing about whether the work it is doing is correct, and that gap is where cascades are born.
Completion monitoring for business flows. It works out the steps a flow should complete from the evidence your systems already emit, and flags the runs that stop halfway while nothing reports an error.
Go · Kafka · OpenTelemetry · WebSocket · early access
A rebalancer for production Kafka, built solo: 17,000+ partition leaders and 50,000+ replicas redistributed in batches of 200, gated by broker disk thresholds. Rebalancing stopped being a maintenance window and became a command.
Python · Kafka internals · Salt · in production · field notes ↗
Designed and implemented end to end: a Go daemon for a service-continuity platform that continuously discovers suspended subscriptions and restores the ones that qualify. Remediation as a daemon rather than a runbook.
Go · Redis · GraphQL · Nomad · in production
First a topic-ownership service, designed and implemented end to end, aggregating ACLs, log dirs and consumer groups. Then, having hit the limits of hand-rolled metadata, DataHub: ElasticSearch reliability and schema-registry integration, contributed upstream to LinkedIn's open source.
Python · Java · ElasticSearch · Confluent Schema Registry
RCS is now widely supported across Android and, since 2024, iPhone: a decade after this work, it has become a mainstream mobile messaging standard, with end-to-end encryption arriving in the Universal Profile. A decade before that, I built the subscriber-data layer such networks need: MySQL, MongoDB and Redis exposed as one OData repository inside a live IMS core. Infrastructure for a standard the world had not adopted yet.
Java EE · OData · MySQL · MongoDB · Redis · Open IMS Core · 11,575 LOC · read the thesis ↗
First P4 implementation of IEEE 802.1CB frame replication, built clause by clause on a nine-switch bmv2 smart grid testbed, with the ONOS P4Runtime driver written to support it. Worked the hardware path on Netronome SmartNICs before running the full testbed on software switches: same proof, none of the cost.
P4 · ONOS · bmv2 · Mininet · SDN · NFV · live testbed
From deep systems mastery to shaping resilient infrastructure, building teams, and crafting tools for others to scale.
Diagnosing, dissecting, and restoring at enterprise scale. The years that proved dashboards can be green while the system lies.
Research conducted the way production is run: live testbeds, working code, and specifications implemented rather than cited.
Delivering working systems without external scaffolding. No frameworks, no teams, often no prior blueprint. Every line of code had to matter.
If you are deep in a system that is failing in ways your dashboards cannot explain, that is the conversation I most want to have. I read every email.