Segun Sogunle

    essay

    Silent Defaulting

    A system cannot tell nobody told me from they told me a legitimate value. So it picks a default, carries on, and produces a wrong answer with no error anywhere.

    Most of what is written about failure is written about the loud kind. A process crashes, a request times out, an error rate climbs and someone gets paged. That literature is mature, and the tools built against it are good.

    This essay is about the other kind, where every component reports success, every dashboard stays green, and the work still did not get done. I have spent several years cataloguing these, first as an operator and now while building against them, and the majority share one mechanism. Once you can name it, you start seeing it everywhere.

    The mechanism

    A system cannot distinguish "nobody told me" from "they told me a legitimate value." So it substitutes a default, proceeds normally, and produces a wrong outcome with no error anywhere.

    Three everyday examples, all of which I have watched cause incidents:

    • An absent field in a JSON document decodes to false in Go. And false is a perfectly reasonable answer to the question "is this key valid?"
    • An unset environment variable reads as the empty string. The empty string is a perfectly reasonable string.
    • A misspelled configuration key is indistinguishable from one deliberately left at its default.

    In every case, absence is coerced into a value that the type system, the parser and the logs all consider legitimate. Nothing fails, because from the machine's point of view nothing went wrong.

    Two properties make this class dangerous out of proportion to how simple it is.

    It fails open, not closed. A crash is loud. A default is silent, and the system keeps running, which means the wrong outcome propagates at full speed to everything downstream.

    Every observability signal reports success. Status codes are 200. Error rates are zero. Latency is healthy. The logs say "validated." Your dashboards are not lying to you. No component failed. The composition failed, and composition is the one thing nothing in the stack is watching.

    A worked example

    A validation service returned its payload at the top level:

    {"isValid": true, "workspaceId": "43e20d08-..."}

    Its caller decoded into a struct expecting the house envelope that every other service in the codebase used:

    {"data": {"isValid": true}}

    There was no data key in the response. The nested struct therefore stayed at its zero value, IsValid read false, and a perfectly valid API key was rejected with HTTP 401.

    The root cause was convention drift. Eight services wrapped their responses in an envelope. One never adopted it. The caller was written against the house standard by a developer who reasonably assumed it held, because it held everywhere else he had looked.

    Here is what the system reported while this was happening:

    A rejected request with every signal green The caller asks the validation service to validate a key. The service replies 200 OK in 3.6ms with isValid true, and logs row_found true, plaintext_match true and Api key validated. The caller's decode succeeds with no error but IsValid reads false, so it rejects the request with HTTP 401. Every metric is green and every log line reports success. Caller Validation service validate key 200 OK, 3.6ms, {"isValid": true} logs: row_found true plaintext_match true "Api key validated" decode succeeds, no error IsValid reads false reject request, HTTP 401 every metric green, every log line success
    Fig. 1 · No component failed. The composition did.

    Walk the signals one at a time:

    • The validation service returned 200. It had done its job correctly.
    • Its own logs read row_found: true, plaintext_match: true, isValid: true, and finally Api key validated.
    • The JSON decode in the caller returned no error. In Go, a missing key is not a decode failure. It yields the zero value.
    • Every access log line in the chain showed status: 200.
    • Metrics would have shown a hundred percent success on the validation service, at 3.6 milliseconds.

    The only detector that worked was a human being trying to use the product.

    There was a compounding factor worth naming separately. Every access log line in the chain carried an empty request identifier. The field existed, so every tool reported the path as instrumented, but it was never populated, so the two sides of the call could not be correlated except by timestamp. A correlation identifier that is present but never populated is worse than none at all, because its presence buys you the belief that you could trace this if you had to.

    Why this is not a bug report

    I could have written that up as one incident and moved on. The reason to generalise is that the same shape keeps arriving wearing different clothes.

    Can the code tell absent from zero A value is absent. If the code can tell absent from zero, the result is a loud contract violation. If it cannot, absence becomes a legitimate value, every component succeeds, the composition produces a wrong outcome, and no signal fires. Value is absent Can the code tell absent from zero? yes no Contract violation, loud absence becomes a legitimate value every component succeeds composition produces a wrong outcome No signal fires
    Fig. 2 · Absence must be representable and loud.

    The provisional taxonomy I keep has two confirmed modes under this parent. Configuration drift, where a value is absent, misspelled or points somewhere stale, and the system falls back to a default and runs. Shape drift, where a consumer reads a field the producer does not emit, the decode succeeds, and a zero value is returned as though it were data.

    Four more are on the list but not yet written up properly, because I have seen them fewer times than I would like before claiming a pattern: ordering drift, where steps execute out of their expected sequence; partial completion, where a chain starts and stops midway with nothing reporting the stop; duplicate execution, where at-least-once delivery meets a step that is not idempotent; and boundary loss, where work in flight when a process is terminated is recorded in neither success nor failure.

    I am deliberately calling the categories provisional. They are hypotheses about how these failures cluster, and the clustering is the thing I am still trying to learn.

    What follows for engineers

    The detection principle is narrow enough to state in one sentence: absence must be representable and loud.

    Wherever a value can be missing, the code must be able to tell missing from present-and-falsy, and must treat missing as a contract violation rather than as an answer. In practice that means pointers or option types rather than bare values at trust boundaries, required-field validation after decoding rather than faith in the decoder, and a response contract that is asserted rather than assumed.

    None of that is novel advice. What is worth internalising is the reason it matters more than it looks. These failures do not degrade your service. They degrade your truth, and they do it while every instrument you own is reporting that everything is fine.

    If you keep an incident log, go back through it and count how many entries were found by a human rather than by a system. In my experience the number is uncomfortable, and the entries have more in common than you would expect.

    Responses

    Threads run on GitHub Discussions. A GitHub account is needed to post. Corrections and counter examples are welcome, particularly from anyone who has run this in production.

    Discuss on GitHubReply by email

    ← All writing