How to tell if your reverse proxy is buffering your token stream (and why I had to rent a Droplet to find out)

If you are streaming tokens from a model to a browser, something in the path can hold them and hand them over in batches. The response still arrives. Every frame is still there. Nothing errors. The only thing that changes is when the bytes show up, and that is the entire user experience of a streaming app.

The advice you will find is to turn off proxy_buffering in nginx. I wanted to check that, so I built a small harness, ran it on a Droplet, and found that nginx was not the problem. HAProxy on a stock config was, and it held the first token for 206 milliseconds.

This is how to measure it yourself. The whole thing is standard-library Python plus Docker, and the Droplet it needs costs about four cents an hour.

The one number you need

Count two things on the client: how many Server-Sent Events frames arrived, and how many recv() calls delivered at least one of them. Divide the first by the second.

Frames per read. 1.0 means every frame arrived on its own, so the stream stayed a stream. 41.0 means all 41 frames landed in a single read, so something in the path buffered the whole response.

It is a ratio, which matters more than it sounds. It needs no baseline to interpret and it survives a noisy host in a way that absolute latency does not.

Before trusting it, point it at something you know buffers. I wrote a ten-line TCP relay that reads the entire upstream response and only then writes it out. Direct it measured 1.00. Through the relay, 21.00. Now you know the instrument can see the thing you are looking for.

Step 1: an emitter you can trust

The measurement is only as good as the thing generating the stream, so the emitter has to be auditable rather than assumed.

Use a plain http.server that sends SSE frames at a fixed interval, and have it do three things:

• Pace against a monotonic deadline computed from a fixed start (start + i * interval), never sleep(interval) in a loop. The second one accumulates drift.
• Set disable_nagle_algorithm = True on the handler. Python leaves Nagle on by default, and small writes waiting on a peer’s delayed ACK will add tens of milliseconds that you will later mistake for a proxy problem. Measured on my Droplet: p90 drift of 10.89ms with Nagle on, 1.29ms with it off.
• Record its own send timestamps and expose them on an endpoint. This is the important one. After each run you can ask the emitter whether it actually paced as instructed, and throw the run away if it did not.

That last point is what separates a measurement from a number. If the emitter stuttered, you want to discard the run, not publish it as a proxy finding.

Step 2: the topology

One emitter, each proxy in its own container pointing at it, and a direct baseline so you always have something to compare against.

services:
  emitter:
    build: ./emitter
    ports: ["127.0.0.1:8080:8080"]

  nginx:
    image: nginx:1.31.5-alpine
    volumes: ["./confdir/nginx.conf:/etc/nginx/nginx.conf:ro"]
    ports: ["127.0.0.1:8081:80"]

  haproxy:
    image: haproxy:3.4.4-alpine
    volumes: ["./confdir/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro"]
    ports: ["127.0.0.1:8085:80"]

Two things worth copying. Pin exact patch versions, because “latest” makes your results unreproducible the moment upstream ships. And bind the published ports to 127.0.0.1, since on a public Droplet 0.0.0.0 exposes an emitter and a set of unauthenticated proxies to the internet for the duration of your run.

Keep the proxy configs boring. Plain reverse proxy, nothing touching buffering. The point is to measure the defaults, so a helpful flush directive you add to one of them quietly answers the question you were asking.

Step 3: the client has to be a raw socket

This is the part people get wrong, myself included at first.

Do not use urllib or requests. They buffer, and client-side buffering contaminates exactly the quantity you are measuring. Open a socket, write the HTTP request by hand, set TCP_NODELAY, send Connection: close so the read loop ends on EOF, and timestamp every recv() the moment it returns.

Then count arrivals per read, not per frame. If two frames arrive in one recv(), that is one arrival carrying two events. Getting that backwards makes every number meaningless while still looking plausible, so write a test that fails if you flip it.

Step 4: certify the Droplet before you trust it

I ran the harness on my laptop first. Do not do this.

Docker Desktop on macOS routes container traffic through a VM network stack, which adds tens of milliseconds of noise to a measurement whose signal is 50 milliseconds. It did not just add error. It manufactured a finding: I measured a consistent 42ms first-token penalty on two proxies, built a mechanism to explain it, and it turned out not to exist. On Linux it vanished in all nine test conditions.

So before measuring anything, have the harness prove the host can measure:

1. Take a direct, unproxied run and require it to show clean incremental arrivals. If a direct connection cannot stream cleanly, no proxy result from that machine means anything.
2. Ask the emitter for its send log and require the pacing to be within tolerance.

Make the tolerance scale with the interval. I used max(2ms, 20% of interval). A flat millisecond budget is meaningless at a 5ms interval — a flat 5ms permits 100% error — and a flat percentage is unachievable there, since 0.2ms is below the scheduling granularity of a shared vCPU.

A s-2vcpu-4gb Droplet running Ubuntu 24.04 certified at 0.22ms of emitter drift. My laptop managed 5 to 6ms, and that is the good case.

What I found

Nine test conditions, ten runs each, four proxies plus a direct baseline. At a 50ms emit interval with roughly 60-byte frames:

nginx, Caddy and Traefik are indistinguishable from having no proxy at all. proxy_buffering is on by default in that nginx row — I confirmed with nginx -T inside the running container rather than trusting the file — and turning it off changes nothing.

HAProxy delivers the frames in bursts of five with zero gap between them, 206ms late.

The part that matters if you are streaming from a model: it is payload-size dependent. Same interval, same stock config, frames grown from about 60 bytes to about 1.1KB:

Grow the payload and it mostly goes away. Token-sized frames are the worst case, and token-sized frames are exactly what LLM streaming produces. Anyone who benchmarked the same proxy with realistic 1KB chunks would never have seen it.

One more result worth knowing: X-Accel-Buffering: no, the header everyone reaches for, does nothing here. 206ms without it, 214ms with it, identical frames per read. It is an nginx convention, and nginx was not the proxy buffering.

Confirming it against real tokens

Everything above uses a synthetic emitter, because exact pacing is what makes the comparison meaningful. But a synthetic emitter is also a fair thing to be suspicious of, so I swapped the upstream for DigitalOcean’s serverless inference endpoint and re-measured through the same client. Fifteen calls against openai-gpt-oss-20b, all HTTP 200:

Synthetic said 5.12 against 1.02. Real tokens said 5.43 against 1.04. The effect is not an artifact of my emitter.

That pass is validation, not proof on its own — a model’s time to first token varies between calls, so the synthetic runs carry the causal argument. But it is a cheap and useful check, and being able to point the rig at a real inference endpoint with one environment variable is what made it a five-minute addition rather than a project.

Caveats I would want if I were reading this

The exact flush trigger in HAProxy is not pinned down. Two manipulations — raising the emit rate and raising the payload size — both reduce the coalescing, which points at filling a buffer rather than a fixed timer. But the byte thresholds implied by different conditions do not reconcile cleanly, around 2.2KB from one and 1.05KB from another, so I would want a dedicated payload sweep before naming a number.

Frames per read has a noise floor around 1.02 on this setup. The direct, unproxied path measures 1.02. So 1.02 for a proxy is not a proxy effect, and reading it as one would be reading noise.

I did not test option http-no-delay, which is the obvious next thing to try on the HAProxy side.

And the exclusions: I discard individual runs whose emitter pacing failed the audit, between 1.7% and 11.7% per condition, because across roughly 2,400 timed writes the odds of one scheduling hiccup approach certainty. That is defensible only because the rule is stated in advance, mechanical, measured on the emitter side independently of the proxy being tested, and every exclusion is counted in the published table. Re-running until the guards pass would fail all four of those, and I know because I tried it: an early run “succeeded” on the sixth attempt, and one honest attempt then certified one condition out of six.

The takeaway I would keep

If you are streaming tokens through a proxy, measure frames per read against a direct baseline before you tune anything. The advice you will find online is aimed at nginx, and in my testing nginx was fine while HAProxy on defaults cost 206 milliseconds of first token.

And measure it somewhere you can trust. The whole exercise cost me about twelve cents in Droplet time, and the same harness on a laptop produced a confident finding that was not real. Four cents an hour is a cheap price for knowing the difference.