Why delay grows like
1 / (capacity − load)

01A server that is fast enough, on average

Take a service that can complete an average of 1,000 requests per second when continuously busy. Its mean service time is one millisecond. Give it 800 requests per second. It has enough capacity: over the long run, it can finish more work than arrives.

It is tempting to conclude that a request should therefore take about one millisecond. That would be true in a carefully synchronized world: one request arrives every 1.25 milliseconds, each takes exactly one millisecond, and the server starts empty. Every request finds the previous one gone.

Now allow arrivals to cluster and service times to vary. Four requests may arrive almost together. Or one request may occupy the server for several milliseconds while more arrive behind it. The server is still fast enough on average. But averages do not let a late request use an idle interval that happened earlier.

A queue is the memory of that mismatch. Surplus work persists until future service clears it. We want to understand how that memory becomes longer as the average arrival rate approaches the service rate—and exactly when the familiar reciprocal formula is justified.

02First, distinguish three different times

Suppose a request arrives at 10 ms. The previous request finishes at 13 ms, so ours starts then. Its own service takes 1 ms, and it leaves at 14 ms. It waited in the queue for 3 ms, received service for 1 ms, and spent 4 ms in the system.

The phrase “delay is 1/(C−L)” hides this distinction. For the M/M/1 model we will derive, the reciprocal is the mean total time in the system, including service. Queueing time alone is smaller by the mean service time. A network's propagation delay, protocol handshakes, and other hops are outside this one-server model.

We also need compatible units. Here capacity and load are requests per second. The reciprocal of their difference has units of seconds per request, interpreted as the mean time associated with one request. If a networking model uses bits per second instead, a packet-size or work-size scale must enter before that reciprocal becomes a packet's delay. Do not silently subtract quantities measured in different units.

03Why spare capacity belongs in the story

Imagine ten requests' worth of extra work has accumulated. In a fluid approximation, with service capacity 1,000 and continuing incoming load 800 requests per second, the backlog drains at a net rate of 200 requests per second. Clearing those ten extra requests takes about 10/200 seconds: 50 ms.

Raise incoming load to 950. The same initial backlog now drains at only 50 requests per second, taking about 200 ms. The machine did not become slower at individual service. New arrivals consume more of the work it completes, leaving less ability to undo an earlier burst.

In first-come-first-served order, later arrivals do not jump in front of the request we are tracking. Their effect on its average delay comes through the backlog that was already present when it arrived. To calculate that backlog, we need a model of variability.

04Make the randomness precise

Let arrivals form a Poisson process with rate λ: disjoint time intervals have independent arrival counts, and interarrival times are exponential with mean 1/λ. Let service times be independent exponential variables with mean 1/μ, independent of arrivals. There is one server, first-come-first-served service, unlimited waiting room, and no drops or abandonment. These are the M/M/1 assumptions.

For our example, μ = 1,000 requests/s. The utilization parameter is ρ = λ/μ. We require λ < μ. The birth-and-death queue treatment in Karl Sigman's notes provides a formal account of this model.

The exponential assumption is useful because the remaining service time has the same distribution regardless of how long the current request has already run. Knowing the number of requests in the system is enough to describe the rates of the next changes. We do not also need to track the current request's age.

Call that number N(t), including any request in service. An arrival raises it by one. A completion lowers it by one, provided it is positive. Unlike a machine with one server per request, our single server does not become four times faster just because four requests are present. Its completion rate while busy remains μ.

05Find how often each backlog occurs

Let π_n be the long-run fraction of time with exactly n requests in the system. Consider the boundary between states n and n+1. In a stable regime, crossings upward and downward must balance in the long run. Every upward crossing must eventually be matched by a downward crossing; the counts can differ by at most one at a particular boundary.

The system is in state n for a fraction π_n of time, and arrivals occur there at rate λ. Downward crossings occur from n+1 at rate μ. Therefore:

This is not a claim that arrivals and departures cancel every instant. It is an accounting identity for long-run crossings. At any particular moment the queue can still surge or empty.

Repeated substitution gives π_n = π₀ρⁿ. All the probabilities must add to one. The geometric series gives 1 + ρ + ρ² + … = 1/(1−ρ), so π₀ = 1−ρ:

At ρ = 0.8, the probabilities of zero, one, and two requests are 0.2, 0.16, and 0.128. Each additional step into a larger backlog multiplies the probability by 0.8. At ρ = 0.95, it multiplies by 0.95 instead: large backlogs become much less strongly suppressed.

To obtain the mean, sum the tail probabilities. A state with N = 3 contributes one to each of the statements N ≥ 1, N ≥ 2, and N ≥ 3. Thus counting those indicators recovers N itself. Taking expectations:

At 800 requests/s, the mean number in the system is four. At 950, it is nineteen. This is where the shrinking capacity margin enters the average backlog, rather than merely an intuition about clearing one chosen burst.

06Turn requests into time

Picture every request's stay as a horizontal interval, from arrival to departure. Add their lengths: you have total request-seconds spent in the system. Now count the same area vertically: at each moment, count how many requests are present and integrate over time. These are two ways of measuring the same accumulated occupancy.

Over a long stable observation, dividing by its duration yields Little's law: the average number in the system equals the throughput times the mean time spent there. Boundary requests need care in a formal proof, but the area accounting is the central idea.

We have earned the reciprocal. In notation where C means service capacity and L means incoming load, this is 1/(C−L). Here L is a rate; it is not the average number of requests, which some books also denote by L. Always map the objects before transferring the formula.

The mean waiting time before service is:

The last five percentage points doubled mean response time. The service-time baseline did not change. This is why a modest admission reduction near saturation can have a large latency benefit in this model.

07Keep the average load. Change the variability.

Use the explorer below to raise utilization from 80% to 95%. The first plot is the exact model curve; the second is a short simulated queue history. Then switch to perfectly regular arrivals and service while keeping the same rates. The server stays just as utilized, but the queue disappears: each request finishes before the next one arrives.

The contrast does not refute the derivation. It changes its assumptions. The rates alone did not specify the distribution of backlog. Perfectly regular arrivals and service at λ < μ produce no waiting from an empty start; the M/M/1 curve is not applicable to that system.

The simulator does not insert the theoretical formula into individual response times. It generates interarrival and service durations, then executes a first-come-first-served recurrence for each request:

You can inspect the simulation source. The curve and numerical simulation arrive at the answer by different routes. Agreement is a useful implementation check, not a replacement for stating the model.

08What happens at full utilization?

In this stochastic infinite-buffer model, ρ = 1 makes the proposed geometric probabilities impossible to normalize. There is no stationary distribution with a finite mean response time. Saying “mean arrivals equal mean service” has removed the net tendency to recover from random excursions; it has not removed the excursions.

Above capacity, the model also lacks a stable stationary regime. A real finite queue cannot grow forever: it drops or rejects work instead. Timeouts, retries, multiple servers, changing service rates, and feedback-driven arrivals can all change the picture. A smooth reciprocal curve is not evidence that a real system satisfies the assumptions behind it.

Notice the contrast with perfectly regular timing: if arrivals occur exactly once per service duration and service is perfectly constant, a carefully aligned single-server system can run continuously without a growing queue. “100% utilization is unstable” must be attached to the stochastic model, not proclaimed as a universal fact about every schedule.

09Take the useful idea into the next system

Suppose the M/M/1 server can be sped up from 1,000 to 1,100 requests/s while arrivals remain 900. Mean service improves from 1 ms to about 0.91 ms. Mean response improves from 10 ms to 5 ms. The larger effect comes from doubling the capacity margin, not merely shaving a tenth of a millisecond from each request.

That is a useful lens for overload control: ask how much recovery capacity remains after admitting new work. When reading a system that adjusts admission, inspect what it measures, how quickly it reacts, and whether its service capacity remains fixed under contention. The reciprocal model supplies a question to investigate, not a guarantee about a particular paper.