What is Server-Sent Events (SSE) streaming for LLM responses?

Short answer: Server-Sent Events (SSE) is an HTTP-based push protocol where a server maintains a long-lived HTTP connection to stream text tokens down to the client in real-time as they are generated by the LLM, eliminating the user perceived latency of waiting for full completions.

Detail

Autoregressive decoding generates text token-by-token. Generating a 500-token response can take several seconds. Without streaming, users face high Time-to-First-Token (TTFT) perceived latency.

How SSE Protocol Works

1. Client Request: Sends HTTP POST/GET request with header Accept: text/event-stream.
2. Server Response: Responds with HTTP header Content-Type: text/event-stream and Transfer-Encoding: chunked.
3. Stream Chunks: Server pushes event data blocks prefixed with data: as each token is sampled.
4. Stream End: Server sends data: [DONE] and closes the HTTP connection.

Client                             Server / vLLM Engine
  │ ────── POST /v1/chat ─────────► (Start Autoregressive Generation)
  │ ◄───── HTTP 200 text/event-stream ───│
  │ ◄───── data: {"token": "The"} ────────│ Token 1
  │ ◄───── data: {"token": " capital"} ───│ Token 2
  │ ◄───── data: [DONE] ─────────────────│ Stream Closed

Example

Python FastAPI endpoint streaming LLM tokens via SSE:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def token_generator():
    tokens = ["The ", "capital ", "of ", "France ", "is ", "Paris."]
    for token in tokens:
        yield f"data: {token}\n\n"
        await asyncio.sleep(0.1) # Simulate decoding step
    yield "data: [DONE]\n\n"

@app.get("/stream")
def stream_llm_response():
    return StreamingResponse(token_generator(), media_type="text/event-stream")

Interview tips

• Compare SSE vs WebSockets: SSE is lightweight, operates over standard HTTP/1.1 or HTTP/2, and is unidirectional (server-to-client), making it ideal for LLM streaming.
• Mention client UI handling: handling chunk buffering, SSE reconnects, and auto-scrolling terminal/chat UIs.

⬅ Back to AI System Design · All topics