Segun Sogunle

    essay

    Why Maintenance Windows Exist, and How to Delete Them

    A maintenance window is a confession: somewhere, an operation was never made bounded, observable, or abortable. What rebalancing 50,000 Kafka replicas taught me about removing the confession.

    Every maintenance window on every operations calendar is a confession. It says: we have an operation whose behaviour under production load we do not trust, so we will perform it when the load, and ideally everyone who would notice, is asleep. The window is not a tool. It is the absence of one.

    I operate production Kafka at a scale where this confession becomes expensive: clusters carrying seventeen thousand partition leaders and over fifty thousand replicas, where storage rebalancing was a scheduled event with a runbook, a window, and a person watching it not finish on time. Within my first two years at that scale I had built the tool that deleted the window, and the design lessons generalise far beyond Kafka. This essay is about why such windows exist at all, and the three properties an operation must gain before its window can be safely removed.

    The anatomy of the window

    Take Kafka partition reassignment as the concrete case, because it is a perfect specimen. Moving a replica between brokers is not a metadata operation. It is a data copy: the destination broker must fetch the partition's log from a live replica, over the same network interfaces and disks that production traffic is using. Ask a cluster to rebalance everything at once, which is what a naive reassignment does, and you have created a second workload the size of your retained data, competing with your first workload, on hardware that was sized for one of them.

    The failure modes compound. Replication traffic saturates the network, so ordinary follower fetching falls behind, so in-sync replica sets shrink, so producers with acks=all stall, so upstream backpressure begins, and now the remedy is causing the disease. Somewhere in the middle a leader election lands on a broker that is busy receiving a terabyte of history. Engineers who have lived one of these nights do not need the mechanism explained; they need it named. The operation is unbounded: its resource consumption is a function of cluster state, not of any limit you set. It is unobservable mid-flight: standard tooling reports that reassignment is in-progress, not whether it is safe to continue. And it is effectively unabortable: cancelling mid-way leaves the cluster in a state you now have to reason about manually, at 3am, from first principles.

    Unbounded, unobservable, unabortable. Any operation with those three properties gets a maintenance window, in any technology, in any decade. The window is how humans contain what the operation will not contain itself.

    Deleting the window is an engineering project, not a policy change

    The tool I built moved those fifty thousand replicas as a background activity, no window, no announcement. Not because it was clever, but because it granted the operation the three missing properties, mechanically:

    The rebalancer control loop The tool observes cluster state, then passes through a health gate. If the gate fails it halts and reports for a human to decide. If it passes, it plans a bounded batch of partitions weighted by disk. A dry run switch either prints the full plan and executes nothing, or moves the batch, verifies completion, elects preferred leaders, and returns to observation. 01 Observe cluster state leaders, replicas, disk per broker nothing is assumed 02 Health gate disk under threshold? ISR stable? metadata consistent? a precondition, checked every pass no Halt. Report. a human decides yes 03 Plan next batch bounded: N partitions, weighted by disk blast radius is the batch 04 Dry run? the same code path produces the plan and the movement plan only Print the full plan execute nothing execute 05 Move batch throttled, abortable 06 Verify completion elect preferred leaders then look again repeat until balanced
    Fig. 1 · A rebalance is not one operation. It is a loop of small, gated, verifiable ones.

    Bounded, by batching. The unit of work was never "the rebalance." It was a batch of at most two hundred partitions, a number chosen so that the worst case, every partition in the batch moving its maximum size concurrently, stayed comfortably inside the headroom of network and disk. The blast radius of any mistake is one batch. Between batches, the cluster is simply a cluster: consistent, serving, in no special mode. There is no window because there is no state that needs one.

    Observable, by gating. Before every batch, not before the operation, the tool re-read the world: per-broker disk utilisation against a threshold, replica distribution, metadata health. The night that made this rule permanent involved a cluster whose brokers stood at 80 to 87 percent disk while a sixth broker waited to take load. The gate refused thresholds it did not like and said so, loudly, per broker, before touching anything. An operation that checks conditions once at start is betting that a multi-hour world holds still. It does not. Transient metadata errors taught the same lesson from the other side: the tool learnt to retry a UnknownTopicOrPartitionException on a ten-second cycle rather than abort a half-planned batch, because at this scale, brokers are always briefly wrong about something.

    Abortable, by construction. Stop the tool between batches and nothing needs cleanup: completed batches are simply done, unstarted ones simply are not. This sounds trivial and is the deepest of the three properties, because it forces the operation to be expressible as a sequence of individually complete, individually verifiable steps. If you cannot express your operation that way, you have found the actual reason your maintenance window exists.

    And underlying all three: the plan is a first-class artifact. The tool's --plan-only flag printed the entire proposed reassignment, per batch, without executing anything. This exists because a cluster will happily execute a plan that is valid and wrong. Validity is the system's to check. Wrongness only shows against intent, and intent belongs to the operator, so the plan must be inspectable by the operator before the system commits to it. Every dangerous operation deserves a renderable plan; most just never get one.

    The general form

    Strip the Kafka specifics and a procedure falls out that applies to schema migrations, storage rebalancing, reindexing, fleet upgrades, and every other window-dwelling operation I have met:

    1. Decompose the operation into steps whose worst-case cost you can state.
    2. Before each step, verify the preconditions in the live world, not in the plan.
    3. After each step, verify completion before the next step is considered.
    4. Make "stopped between steps" a fully valid, fully described system state.
    5. Render the whole plan for human inspection before the first step runs.

    None of this is exotic. All of it is work, which is why the calendar invite is so tempting: a window costs a meeting, and the properties cost engineering. But the window's price recurs forever, compounds with scale, and concentrates risk into the hours when your best people are at their worst. The engineering pays once.

    The wider point, and the reason this essay sits beside my writing on observability: a maintenance window is the operational symptom of a system that cannot verify its own state mid-change. The batch gate asking "is the world still safe?" is the same question as the reconciliation check asking "did the data all arrive?" Both are statements of intent, held up against reality, continuously. Systems that can answer such questions do not need the building empty before they act.

    Windows are for buildings. Systems should keep working with the lights on.

    Responses

    Threads run on GitHub Discussions. A GitHub account is needed to post. Corrections and counter examples are welcome, particularly from anyone who has run this in production.

    Discuss on GitHubReply by email

    ← All writing