Why Your LLM Token Stream Arrives All At Once in Production

AI Tooling · Intermediate · 6 min read · published 2026-08-14

What this solves: Server-sent-event or chunked streaming endpoints that emit tokens smoothly on localhost deliver one giant blob after 20 seconds once deployed behind a proxy or CDN. This explains where the chunks are being held and how to force them through.

The Problem

Your chat endpoint streams model output as Server-Sent Events. On your laptop it's beautiful: first token in 180ms, words appearing at ~40/sec. You deploy behind the standard nginx ingress and the UI goes dead for 17.4 seconds, then paints the entire 900-token answer in a single frame.

Nothing is broken. The bytes are correct. Logs show the app yielded 900 chunks, the first at t=0.2s. The browser's DevTools network panel shows one response with 900 events, starttransfer at 17.3s. Perceived latency went from "instant" to "is this thing frozen?", and your abort/stop button now cancels a request whose output the user already never saw arriving.

Why the Obvious Fix Falls Short

The first instinct is "the proxy is timing out or the connection is slow" — so people bump proxy_read_timeout, raise keepalive_timeout, add proxy_http_version 1.1. None of that changes anything, because the problem isn't the connection lifetime; it's that an intermediary is accumulating your body before forwarding it.

The second instinct is more expensive: "SSE must be flaky, let's move to WebSockets." That's a multi-day rewrite (auth, reconnection, backpressure, sticky sessions) that fixes the symptom only accidentally — WebSockets bypass response buffering because they're not HTTP responses at all. But you've now taken on a stateful protocol, and the same class of bug reappears as CDN/ALB upgrade-header handling and idle-timeout drops.

The third instinct — calling flush() harder in the app — fails because your app is already flushing. The flush pushes bytes into the socket to nginx; nginx happily accepts them into proxy_buffers and sends nothing downstream until the buffer fills or the upstream closes. A 900-token answer is maybe 4KB; the default buffers are 8KB. So it never fills, and the whole thing ships at EOF. That's exactly why the delay equals your total generation time, not some fixed interval — a strong diagnostic signal.

Compression is the sneaky twin. gzip on in nginx (or Cloudflare's auto-minify/compression, or Flask-Compress) needs enough input to produce a deflate block. Even with buffering off, a gzip filter can hold your 20-byte chunks indefinitely.

How It Actually Works

Streaming is a property of the entire chain, and every hop can independently buffer. A chunk is only visible to the user when every hop has chosen to forward it.

nginx's rule is simple: if proxy_buffering is on, it reads from upstream into buffers and writes downstream when a buffer is full, when proxy_busy_buffers_size allows, or at EOF. The escape hatch is the X-Accel-Buffering: no response header — nginx strips it and switches that single response into unbuffered mode. This is better than a global proxy_buffering off because normal JSON endpoints still benefit from buffering (it frees the upstream worker sooner and shields slow clients).

Downstream of nginx, CDNs apply their own rules keyed on content type and Cache-Control. text/event-stream is usually passed through; application/json or text/plain often isn't. Cache-Control: no-transform tells well-behaved intermediaries not to recompress.

sequenceDiagram
    participant M as Model
    participant A as App (ASGI)
    participant N as nginx
    participant C as CDN
    participant B as Browser
    M->>A: token "Hel"
    A->>N: chunk (flushed, 20B)
    Note over N: proxy_buffering on<br/>8KB buffer not full → hold
    M->>A: token "lo"
    A->>N: chunk
    Note over N: still holding
    M--xA: generation done (t=17s)
    A->>N: EOF
    N->>C: all 4KB at once
    Note over C: gzip filter also<br/>needs a full block
    C->>B: single blob at t=17.4s
    Note over B: user saw nothing<br/>for 17 seconds

The mental model: treat each hop as a valve with its own "forward now?" predicate. Your job is to make every predicate say yes per chunk — via unbuffered mode, no transformation, and no Content-Length. That last one matters: anything that computes Content-Length must have consumed the whole body first, so a Content-Length on a streaming response is proof something buffered it.

Before and After

# BEFORE: streams locally, blobs in prod
@app.get("/chat")
async def chat(q: str):
    async def gen():
        async for tok in llm.stream(q):
            yield f"data: {json.dumps({'t': tok})}\n\n"
    # media_type defaults to text/plain; no anti-buffering hints;
    # GZipMiddleware is installed app-wide and will hold small chunks
    return StreamingResponse(gen())

# AFTER: explicit content type, unbuffered hints, keep-alive pings,
# and gzip excluded for this route
@app.get("/chat")
async def chat(q: str):
    async def gen():
        last = time.monotonic()
        async for tok in llm.stream(q):
            yield f"data: {json.dumps({'t': tok})}\n\n"
            last = time.monotonic()
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",   # CDNs pass this through
        headers={
            "X-Accel-Buffering": "no",    # nginx: unbuffer THIS response
            "Cache-Control": "no-cache, no-transform",  # no recompression
            "Connection": "keep-alive",
        },
    )
# plus: GZipMiddleware(minimum_size=..., excluded paths) or drop it and let
# the edge compress non-streaming routes only

When NOT to Use This

• Short responses. If the full answer is 300ms of generation, streaming machinery buys nothing and costs you connection-held-open time. Just return JSON.
• You need durable resumption. SSE reconnects re-run your handler; if a dropped connection must not re-charge you for tokens, you want a job + polling model (or SSE over a server-side buffer keyed by request id), not raw pass-through streaming.
• API Gateway (REST) / some serverless fronts. They fully buffer responses by design; no header saves you. Use a function URL with response streaming, an ALB, or WebSockets.
• Global proxy_buffering off. Don't. Unbuffered responses pin an upstream worker to the slowest client on the internet. Scope it per-route.

Gotchas

• A Content-Length header on your streaming response is the smoking gun that something buffered it — usually a compression or logging middleware.
• curl lies. Without -N, curl buffers stdout and you'll "reproduce" a bug that isn't there — or miss one that is.
• Idle timeouts kill silent streams. A model that thinks for 70s before the first token trips Cloudflare's ~100s and many ALB 60s idle timeouts. Emit : keep-alive\n\n comments every 10–15s.
• HTTP/1.0 to upstream disables chunked encoding. If you didn't set proxy_http_version 1.1, nginx must buffer to know the length.
• EventSource auto-reconnects forever, including after your server returns a 500. Without id:/Last-Event-ID handling or an explicit terminal event, one bad deploy becomes a self-inflicted retry storm against a paid inference API.
• Browser sniffing: streaming text/html can be withheld until ~1KB arrives for charset detection. Another reason to use text/event-stream.
• Service meshes count too. Envoy/Istio sidecars generally stream fine, but a WASM or Lua filter that inspects the body will buffer it whole.

Key takeaway: Streaming is only as incremental as the least-flushing hop in the chain: set `X-Accel-Buffering: no`, disable proxy buffering and compression for `text/event-stream`, and never let anything add a Content-Length.

# 1. app directly (inside the pod)
curl -N -w '\n%{time_starttransfer}\n' http://localhost:8000/chat
# 2. through nginx service
curl -N http://ingress.internal/chat
# 3. through Cloudflare
curl -N https://app.example.com/chat

return Response(
    generate(),                      # yields "data: ...\n\n" chunks
    mimetype="text/event-stream",
    headers={
        "X-Accel-Buffering": "no",   # nginx: skip response buffering
        "Cache-Control": "no-cache, no-transform",  # no-transform stops CDN gzip
        "Connection": "keep-alive",
    },
)