Why Breakwater deliberately overcommits

01The server that drowns in cheap work

Imagine an RPC server whose ordinary request takes 10 μs. That sounds like abundance: one core could complete roughly 100,000 such requests per second if every request really cost 10 μs and overhead vanished. But it also means that an overload controller has almost no budget of its own. A control exchange that costs a few microseconds is no longer cheap beside the work it regulates.

Now connect 10,000 clients. Most are quiet; any subset may wake at once. If each client is allowed even one unconditional outstanding request, all 10,000 can arrive together. The server’s queue becomes the place where the clients discover that they collectively asked for too much.

That is a particularly bad place to learn the news. By the time a request reaches the head of a long server queue, its latency objective may already be impossible. The server has also paid network and packet-processing costs for work it may eventually reject. Feedback sent after overload must wait behind the overload it is trying to stop.

This seems to solve overload by construction. Count the amount of work the server can tolerate, create exactly that many credits, and never admit more. But “create exactly enough” is only half the problem. The other half is getting the credits to clients that currently have work.

02One credit ledger, followed all the way through

Use a toy time slice. The server can begin four requests during it. There are four clients—A, B, C, and D—and the server’s last information says each wants one request. It therefore places one credit at each client.

Between that report and the credits’ arrival, demand changes. A now has four requests; B, C, and D have none. Only A’s credit is used. Three permissions exist, but they are stranded at clients with no work. The server begins one request and leaves three service slots idle.

No queue formed. Latency looks wonderfully controlled because useful work was prevented from arriving. Exact admission protected the server from overload by risking the opposite failure: underload.

Why not ask every client for fresh demand before every grant?

Because there are 10,000 clients and the RPC itself lasts microseconds. Explicit demand updates and explicit grants can consume the very packet-processing capacity the controller is meant to preserve. An extra permission round trip can also dominate an RPC’s service time. The accurate solution changes the workload.

Breakwater therefore learns demand opportunistically. Clients piggyback demand on requests they are already allowed to send; the server piggybacks credits on responses it is already returning. A newly registering client can send its first request and demand together. This removes most dedicated coordination messages, but it makes the server’s demand view stale by design.

03The unsettling repair: promise the same capacity more than once

Return to the ledger. Suppose the server permits each client to retain one additional speculative credit. A now holds two credits and can send two of its four requests. The other three speculative credits are still stranded, but utilization rises from one service slot to two.

If measured queueing delay remains below target, the controller increases the global credit pool again at the next update. Repeated positive evidence—“the server is not yet queueing enough”—pushes more permissions into circulation until active clients collectively deliver enough work. Breakwater calls this combination demand speculation with credit overcommitment.

The word overcommitment is literal. The server can issue more credits than its immediate capacity because it predicts that not every client will exercise every permission at once. The credits are not a conservation law. They are a controlled wager on sparse, stale demand.

Try the two corners. With only A active and no speculative credits, three credits are safe but stranded. Raise the speculative credits until A can fill the server. Then wake all four clients without changing the credits. The same speculation that rescued utilization now permits an incast: more requests arrive than this round can serve.

This is not a bug in the toy or an accidental corner of the paper. It is the central trade: information avoided at the control plane reappears as risk at the data plane.

04How the feedback loop sizes the wager

Breakwater does not use the toy’s fixed four slots. It maintains a global credit pool, C_total, intended to represent the received load the server can handle while meeting its latency objective. Once per network RTT, it compares measured queueing delay d_m with a target d_t.

Below target, the controller probes upward additively. Above target, it backs off multiplicatively, with the reduction scaled by how far delay exceeded the target and capped so one update does not remove more than half the pool. The paper sets the additive step A proportional to the number of clients: A = max(αn_c, 1). That is how the controller’s aggressiveness scales with the population over which speculative credits may be stranded.

This resembles AIMD, but do not transfer the network picture mechanically. The controlled object is a server-wide permission pool, the congestion signal is request queueing delay, and the resource may be CPU, disk, or another bottleneck. Individual clients are not independently estimating a fair path share.

When the pool grows, the server issues credits—preferably on outgoing responses. When it shrinks, it stops issuing more or sends negative credits to revoke credits that clients have not consumed. A request already sent cannot be unsent. That is why the loop can reduce future load but still needs a defense against the burst already in flight.

05Why queueing delay is the signal

CPU utilization misses non-CPU bottlenecks. A disk-bound RPC can wait while CPU utilization looks low. Queue length is also ambiguous when service times vary: two queued 1 μs requests and two queued 100 μs requests have the same length but very different latency consequences.

Queueing delay measures the symptom the SLO actually budgets. Breakwater timestamps queued items and uses the age of the oldest item. It sums delay across two stages: packets waiting to become requests, and request threads waiting to execute. This keeps the feedback tied to accumulated waiting rather than one resource proxy.

There is a subtle dependency here. Delay is only a useful overload signal if unrelated scheduling artifacts do not dominate it. Head-of-line blocking behind one long request can look like overload even when more capacity would not solve the real problem. Breakwater therefore uses a dispatcher model and is implemented over Shenango’s low-latency stack and lightweight threads. The substrate is not just a performance footnote; it helps make the control signal interpretable.

06Credits prevent the common case; AQM catches the mistake

At this point the design has created a contradiction on purpose:

The last risk is handled with delay-based active queue management. If queueing delay crosses an SLO-derived threshold, the server drops requests. This is not the primary controller: the credit mechanism is supposed to prevent most excess requests from reaching the server. AQM is the safety valve for the occasional incast created by speculation.

That layering matters. If every overload episode were handled only by dropping at the server, clients would receive slow feedback and the server would repeatedly spend work on doomed requests. If credits were perfectly conservative, utilization could collapse whenever demand knowledge became stale. Breakwater uses a cheap imperfect predictor for the common case and a sharper costly action for prediction errors.

Clients also enforce the end-to-end objective. Requests waiting locally expire according to their SLO; moving a queue from server to client does not make its time disappear. It makes admission controllable and lets expired work die before consuming server resources.

07Return to the opening burst

Ten thousand sporadic clients defeat the comforting idea that “one outstanding request per client” is a small window. The global minimum becomes 10,000. Server-driven credits reverse the authority: zero is a valid client window until the server grants access.

But the server cannot know every client’s current demand without paying for that knowledge. Our four-credit ledger showed the consequence. Exact but misplaced permission left 75% of the server idle. Overcommitment put enough permissions in circulation to find the active client, at the cost of a possible burst when all clients woke together. Queue-delay feedback adjusts the size of the wager; AQM limits its worst short-term error.

The transferable insight is broader than RPC overload control:

Change one assumption: suppose demand is dense and synchronized rather than sparse and sporadic. Almost every client exercises every credit immediately. Overcommitment no longer hides stranded capacity; it mostly manufactures bursts. A less aggressive setting would trade some goodput for fewer drops. The mechanism stays the same, but the workload changes which side of its trade is valuable.