SGLang

Part I: The mental model

HTTP request (OpenAI JSON, schema attached)
      │
      ▼
FastAPI server                            srt/entrypoints/http_server.py
      │
      ▼
TokenizerManager: tokenize, dispatch      srt/managers/tokenizer_manager.py
      │   (ZMQ queue to the scheduler process)
      ▼
Scheduler loop                            srt/managers/scheduler.py
      ├─► radix tree: longest cached prefix?   srt/mem_cache/radix_cache.py
      ├─► grammar: compile schema → automaton  srt/constrained/
      ├─► build batch                          srt/managers/schedule_batch.py
      ▼
TP worker ──► model runner ──► attention  srt/managers/tp_worker.py,
      │        backend kernel              srt/model_executor/, srt/layers/attention/
      ▼
grammar bitmask ──► sample one token      srt/layers/sampler.py
      │        (or jump-forward: append many tokens, no forward pass)
      ▼
DetokenizerManager: ids → text            srt/managers/detokenizer_manager.py
      │   (ZMQ back to TokenizerManager)
      ▼
SSE chunk to the client; KV pages stay indexed in the radix tree

The one-sentence identity: SGLang is a serving runtime organized around reuse: a radix tree that makes every past computation a candidate for sharing, grammars that skip tokens the output format has already determined, and a frontend language whose programs expose that structure on purpose. Where vLLM begins from memory allocation and adds caching as a feature, SGLang begins from the observation that real workloads repeat themselves, and builds the runtime around exploiting the repetition.

Three ideas carry the design. RadixAttention keeps the entire KV cache pool indexed by a radix tree over token sequences, so a new request's prompt is first matched against everything the server has ever computed and still holds, and only the unmatched suffix is prefilled. Constrained decoding compiles a JSON schema, regex, or grammar into an automaton that masks invalid tokens each step, and, crucially, fast-forwards through stretches where the grammar permits exactly one continuation, so the model only ever runs forward passes where it has a real choice. And the frontend DSL, the Structured Generation Language the project is named for, lets callers write whole multi-call programs whose branching and sharing the runtime can see and schedule for, instead of inferring structure from a stream of isolated requests.

Operationally, the runtime (called SRT, the SGLang Runtime) is a pipeline of processes connected by message queues: an HTTP process that tokenizes, a scheduler process that owns the GPU, and a detokenizer process, so Python string work never blocks a forward pass. Keep that pipeline in your head; every stage in Part IV is one hop along it.

Part II: Using it

Installing

The runtime targets Linux. On an NVIDIA GPU box with Python 3.10+:

pip install sglang
# the docs recommend uv for speed and correct pins:
uv pip install sglang

The wheels pull in a matched CUDA PyTorch plus the sgl-kernel package of custom kernels; when you hit version friction (usually a PyTorch/driver mismatch), the install guide's pinned combinations and the official lmsysorg/sglang Docker images are the reliable path, and Docker is what the project recommends for production. AMD ROCm and CPU backends exist behind their own extras. On macOS you can install the package for the frontend language and clients, but the serving runtime itself needs Linux; point the DSL at a remote server instead.

First real session

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-0.5B-Instruct \
  --host 0.0.0.0 --port 30000

Startup logs show weight loading, KV cache pool sizing (derived from --mem-fraction-static, which is auto-computed by default), and CUDA graph capture, then the server listens on port 30000. Once running, every request prints scheduler telemetry shaped like this (wording varies by release):

Prefill batch. #new-seq: 1, #new-token: 42, #cached-token: 0, ...
Decode batch. #running-req: 1, #token: 60, token usage: 0.00, gen throughput (token/s): ...

That #cached-token field is RadixAttention speaking, and you will use it constantly. The server is OpenAI-compatible, so the usual client works with a base URL change:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)

Structured output

The signature feature. Attach a JSON schema through the OpenAI structured-outputs surface and the server guarantees the response parses, enforced at the token level by the grammar backend (XGrammar by default, verified in the v0.5.15 server args):

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Extract: Ada Lovelace, born 1815."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "born": {"type": "integer"}},
                "required": ["name", "born"],
            },
        },
    },
)

The native /generate endpoint accepts the same constraint directly in sampling_params (json_schema, regex, or an EBNF grammar), which is handy for scripting outside the OpenAI shapes.

The frontend DSL

The same package contains the language the project is named for, a Python-embedded DSL that runs against the server you launched:

from sglang import function, gen, system, user, assistant, RuntimeEndpoint
from sglang.lang.api import set_default_backend

set_default_backend(RuntimeEndpoint("http://localhost:30000"))

@function
def basic_qa(s, question):
    s += system("You are a helpful assistant.")
    s += user(question)
    s += assistant(gen("answer", max_tokens=512))

state = basic_qa("List 3 countries and their capitals.")
print(state["answer"])

The mistakes beginners make

The classic one is defeating the radix cache with prompt ordering. The tree shares prefixes, so any dynamic content placed before the static content splits the tree at token one:

# Wrong: timestamp first → zero prefix reuse across requests
prompt = f"[{datetime.now()}] {LONG_SYSTEM_PROMPT}\n{question}"

# Right: static prefix first, dynamic content last
prompt = f"{LONG_SYSTEM_PROMPT}\n{question}\n(asked at {datetime.now()})"

The same applies to few-shot examples (keep them identical and leading) and to chat histories (append, never rewrite earlier turns). The second mistake is benchmarking with a sequential loop; like every continuous-batching engine, SGLang shows its throughput only under concurrency, so use an async client and asyncio.gather. The third is assuming a schema makes output correct rather than parseable: constrained decoding guarantees syntax, and a model can still hallucinate a perfectly valid wrong answer into it. The fourth is reading token usage panic into normal behavior: a nearly full KV pool is often just the radix cache holding warm history, which eviction reclaims on demand.

Part III: When it is the right tool

SGLang shines exactly where its ideas bite: multi-turn chat and agent loops that resend growing histories, few-shot evaluation sweeps where thousands of requests share their examples, heavy structured-output pipelines (extraction, tool calls, JSON APIs), and large multi-replica deployments where its disaggregated prefill/decode serving and cache-aware routing were built at real scale. If your traffic has high prefix overlap or is mostly schema-constrained, it is the strongest choice.

The alternatives: vLLM overlaps almost completely, has the larger ecosystem and broadest day-one model and quantization support, and is the safer generic default; the two projects have converged hard (vLLM has prefix caching and XGrammar too; SGLang has paged memory and chunked prefill), so the choice is often benchmarks on your own workload. TensorRT-LLM trades flexibility for peak NVIDIA performance with ahead-of-time builds. llama.cpp and Ollama own single-user local inference, where a scheduler built for thousands of concurrent requests is dead weight.

The architecture-shaped warning: RadixAttention's value is destroyed one layer above it, at the load balancer. Each replica has its own radix tree; a cache-blind round-robin sprays a conversation's turns across replicas, so every turn misses:

Dangerous:                          Safe:
turn 1 ─► replica A  (miss)         turn 1 ─► replica A  (miss)
turn 2 ─► replica B  (miss!)        turn 2 ─► replica A  (hit)
turn 3 ─► replica C  (miss!)        turn 3 ─► replica A  (hit)
   round-robin LB                     prefix-aware / sticky routing

The project ships its own answer, a Rust model gateway (sgl-model-gateway/ at the repo root, descended from the sgl-router project) that routes with an approximate view of each replica's cached prefixes. If you front SGLang with your own balancer, you need at least session stickiness, or you are paying for a cache you never hit.

Part IV: The full life of one request

The canonical operation is one structured-generation request: a chat completion carrying a JSON schema, streamed back. Paths are python/sglang/-relative where obvious, verified against v0.5.15.

Stage 1: HTTP arrival

launch_server.py boots the FastAPI app in srt/entrypoints/http_server.py; the OpenAI-compatible routes live in srt/entrypoints/openai/ and translate the request into the internal generate form: chat template applied to the messages, the schema captured as a constraint in the sampling parameters. (There are also gRPC and offline Engine entrypoints beside it; all converge on the same next stage.)

Stage 2: TokenizerManager

The TokenizerManager (srt/managers/tokenizer_manager.py) lives in the HTTP process. It tokenizes the prompt, wraps everything into a tokenized request message (the message types are all in srt/managers/io_struct.py, a surprisingly clarifying file to read early), sends it over a ZMQ queue to the scheduler process, and keeps the mapping needed to stream results back to the right HTTP response later.

Stage 3: Grammar compilation

Because the request carries a schema, the scheduler routes it through the grammar machinery in srt/constrained/ before it can run: the backend (XGrammar via xgrammar_backend.py by default; Outlines and llguidance are alternatives behind base_grammar_backend.py) compiles schema to grammar to automaton. Compilation can take real milliseconds, so compiled grammars are cached by key and the request simply waits in a grammar queue on a cold miss while other requests keep the GPU busy. Every later request with the same schema skips this entirely.

Stage 4: Radix match and admission

The scheduler (srt/managers/scheduler.py) runs the engine's event loop. For our waiting request it asks the radix cache (srt/mem_cache/radix_cache.py) for the longest cached prefix of the prompt's token IDs. Whatever matches, a shared system prompt, the previous turns of this conversation, is reused directly: the matched KV pages are locked against eviction and only the unmatched suffix needs prefill. The admission logic in srt/managers/schedule_policy.py then packs a prefill batch under token and memory budgets (long suffixes are chunked, like vLLM's chunked prefill), and the scheduling policy orders the waiting queue: the current release defaults to fcfs, with lpm, longest-prefix-match first, as the classic cache-aware policy from the paper that groups prefix-sharing requests together to keep hit rates high.

Stage 5: Batch formation and the forward pass

srt/managers/schedule_batch.py turns the chosen requests into a batch, allocating KV pages from the token pools in srt/mem_cache/memory_pool.py and srt/mem_cache/allocator/. The batch crosses to the tensor-parallel worker (srt/managers/tp_worker.py) and the model runner (srt/model_executor/model_runner.py, with per-batch metadata in forward_batch_info.py), which executes the model with an attention backend chosen from srt/layers/attention/ (FlashAttention and FlashInfer backends among many; custom kernels live in the separate sgl-kernel/ package at the repo root). Decode steps replay captured CUDA graphs, and by default the scheduler runs an overlap mode that prepares the next batch on CPU while the GPU executes the current one, hiding Python time entirely.

Stage 6: Constrained sampling, or no sampling at all

The forward pass yields one logits row for our request. Before sampling, the grammar automaton supplies a bitmask of legal next tokens, applied to the logits so illegal tokens get probability zero; then the sampler draws normally (temperature, top-k/top-p). The automaton advances on the chosen token. And here is the fast path: when the automaton reports that only one string can follow, for example ", "born": after a closed name field, the runtime appends those tokens directly, without any forward passes, and resumes decoding at the next real choice. That is jump-forward decoding, and it is why rigid schemas get faster while pure free-text does not.

Stage 7: Detokenize and stream

Output token IDs go over ZMQ to the DetokenizerManager (srt/managers/detokenizer_manager.py), which performs incremental, stateful detokenization (BPE pieces are not characters; it buffers until clean UTF-8 and stop conditions can be checked), then sends text deltas back to the TokenizerManager, which resolves them to the waiting HTTP response as SSE chunks.

Stage 8: The cache remembers

When the request finishes, its KV pages are not freed in the vLLM sense; the token sequence, prompt plus generated output, is inserted into the radix tree and the pages stay resident, unlocked and evictable LRU. The next turn of this conversation will match almost its entire prompt. This is the philosophical signature of the system: completion does not free memory, it publishes the computation for reuse.

Part V: Internals deep dives

RadixAttention and the radix-tree prefix cache

A radix tree is a compressed trie: edges carry token sequences, not single tokens, and nodes split only where stored sequences diverge. SGLang keys the tree on token IDs, and each node maps its edge's tokens to their KV page indices in the pool:

                    root
                     │ [system prompt, 1200 tokens]
                     ●
          ┌──────────┴───────────┐
   [chat A turn 1]         [chat B turn 1]
          ●                       ●
     ┌────┴────┐                  │ [chat B turn 2]
[A turn 2] [A turn 2']            ●
   (two sampled branches)

match_prefix(new prompt) → walks from root, returns
longest shared prefix's KV indices + the node to lock

Three operations define it. match_prefix walks the tree along a new prompt and returns the reusable KV; insertion after a request finishes extends or splits nodes (a divergence in the middle of an edge splits it, exactly like a trie); and eviction walks leaves in LRU order, freeing pages of branches nobody has touched, but never nodes whose lock_ref is held by a running request. The important accounting fact, and the misconception to kill: the tree stores no tensor data; it is an index over the one shared KV pool, so "the radix cache" costs metadata, not duplicate KV memory. A full-looking pool is mostly evictable history, and the /flush_cache endpoint empties it on demand.

Granularity is a real design difference from vLLM: SGLang's KV allocation page size defaults to a single token on CUDA (--page-size, default resolved to 1, verified in the v0.5.15 source), so the tree can share prefixes to token precision, where vLLM's block hashing shares only whole 16-token blocks. Larger pages are supported and some attention backends require them; the trade is coarser sharing for friendlier memory layout. Around the core sit the production variants you should know exist before reading srt/mem_cache/, because the directory is crowded: chunk_cache.py is the degenerate no-sharing cache used when the radix cache is disabled (--disable-radix-cache), and hiradix_cache.py plus hicache_storage.py extend the tree hierarchically so evicted-from-GPU prefixes can live in CPU RAM or remote storage and be pulled back instead of recomputed.

Cache-awareness closes the loop with scheduling: because the tree makes reuse visible before running anything, the scheduler's lpm policy can sort waiting requests by matched prefix length so requests sharing prefixes run together, raising hit rates under load; the paper reports large throughput wins from this on shared-prefix benchmarks. Note honestly that the current release defaults to plain fcfs (predictable ordering, priority support), with lpm one flag away, and that under memory pressure the scheduler retracts running requests, SGLang's word for preemption, tunable via --schedule-conservativeness.

Constrained decoding: masks, compressed FSMs, jump-forward

Baseline grammar-constrained decoding compiles the constraint to an automaton over tokens and, each step, masks the logits so only legal tokens survive. Correct, but it still pays one full forward pass per token, even for tokens the grammar has already decided. SGLang's contribution (introduced as jump-forward decoding with a compressed finite state machine, in the paper and the early 2024 blog post) starts from a simple observation about automata: runs of states with exactly one outgoing edge are stretches where the model has no choice, so they can be compressed into one edge and emitted in one step.

Schema: {"name": string, "born": integer}

{"name": "  A d a  " , " b o r n " :  1 8 1 5 }
└───┬────┘ └─┬─┘ └──────┬───────┘ └─┬──┘
 jump       model      jump        model
 (0 passes) decodes    (0 passes)  decodes

For a rigid schema most of the output is punctuation and fixed keys, so most tokens cost zero forward passes. One subtlety makes the implementation honest rather than naive: jumped text is appended as a string and the boundary re-tokenized, because tokenizers merge across the seam (the last generated token plus the jumped text may tokenize differently than their concatenation), and getting this wrong corrupts the KV cache's correspondence to the text. The retokenization logic sits with the jump-forward map in srt/constrained/outlines_jump_forward.py for the regex/FSM backend.

The default engine, though, is XGrammar (srt/constrained/xgrammar_backend.py; default confirmed in the v0.5.15 server args), which generalizes from regular expressions to context-free grammars with a pushdown-automaton design: it precomputes token bitmasks for context-independent automaton states at compile time and computes only the context-dependent remainder at runtime, overlapping that CPU work with the GPU forward pass, and it exposes its own jump-forward strings. Outlines and llguidance remain as alternative backends behind the common base_grammar_backend.py interface. Two corrections worth internalizing: constrained decoding does not slow generation down in any fundamental way (with mask precomputation it is near-free per step, and jump-forward makes rigid formats faster than unconstrained); and it does subtly reshape the distribution, because masking renormalizes among legal tokens, so a schema guarantees syntax, not the answer the unconstrained model "would have" given.

The frontend DSL and the co-design argument

SGLang stands for Structured Generation Language, and the language half (python/sglang/lang/) is the original point of the paper: if callers describe whole programs, the runtime can see structure it could never infer from isolated requests. A decorated function builds an intermediate representation (lang/ir.py) executed by an interpreter (lang/interpreter.py) that manages a per-call prompt state asynchronously against a backend (lang/backend/, with RuntimeEndpoint speaking the server's native API). Three primitives do the work: gen appends a generation slot and is non-blocking, so Python only waits when you read the value; select picks among fixed options by scoring each continuation's probability under the model (the normalization choices live in lang/choices.py) instead of free-decoding and parsing; and fork splits one state into parallel branches:

@function
def compare(s, topic):
    s += user("Analyze " + topic + " from three angles.")
    forks = s.fork(3)
    for f, angle in zip(forks, ["cost", "risk", "speed"]):
        f += assistant(gen("view", max_tokens=128, stop="\n"))
    s += assistant("Summary: " + gen("summary", max_tokens=128))

The co-design is that each primitive maps to a runtime strength: fork's branches all share the parent prompt's KV through the radix tree, so three branches cost one prefill; select is a few cheap scoring passes over cached prefix; interleaved gen calls across many program instances keep the continuous batch full. Honesty requires saying that most production users drive SGLang through the plain OpenAI API and the DSL is the less-traveled half of the project. But it explains the architecture: the runtime is built to exploit exactly the sharing and parallelism the language expresses, and the radix tree serves API users precisely because it was designed for programs shaped like the one above.

Part VI: Reading the repository

Almost everything lives under python/sglang/, split between lang/ (the frontend) and srt/ (the runtime, where to spend your time). The tree has grown a lot of production surface (disaggregation, hierarchical cache, speculative decoding, many mixins), so read on rails:

Stage 0, orientation (one evening). Read the SGLang paper (Zheng et al., NeurIPS 2024) and the jump-forward blog post. Questions: what does co-design mean concretely? Why a radix tree rather than a hash map of prefixes?

Stage 1, the skeleton. Read python/sglang/launch_server.py, srt/entrypoints/http_server.py, and skim srt/entrypoints/engine.py and srt/managers/io_struct.py. Questions: which processes exist and what flows between them? Where does the OpenAI layer end and the native API begin?

Stage 2, the pipeline. Read srt/managers/tokenizer_manager.py and srt/managers/detokenizer_manager.py. Questions: why is detokenization its own process? What state does each side keep per in-flight request?

Stage 3, the heart. Read srt/managers/scheduler.py (long; read the event loop first), then schedule_policy.py and schedule_batch.py. Questions: walk one iteration of the loop. What is a retraction and when does it fire? How does lpm ordering differ from fcfs?

Stage 4, the memory. Read srt/mem_cache/radix_cache.py alongside memory_pool.py, then skim chunk_cache.py as the contrast case. Questions: what do match_prefix, insert, and evict do to the tree? What does lock_ref protect against? Where is the actual KV tensor?

Stage 5, compute and constraints. Read srt/model_executor/model_runner.py and forward_batch_info.py, one backend in srt/layers/attention/ (flashinfer_backend.py or flashattention_backend.py), then srt/constrained/base_grammar_backend.py and xgrammar_backend.py. Questions: how does a batch describe itself to the kernel? Where exactly does the bitmask meet the logits? Where does jump-forward re-enter the scheduler?

Stage 6, the language. Read lang/api.py, lang/ir.py, and lang/interpreter.py. Questions: what does fork actually send to the server? Why is gen lazy?

Where not to start: the disaggregated prefill/decode machinery, sgl-kernel/, speculative decoding, the hierarchical cache variants, and the swarm of *_mixin.py files. All are real production engineering layered on the core loop, and none make sense before Stages 3 and 4 are solid.

Part VII: Hands-on labs

All labs assume a Linux GPU box running python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-0.5B-Instruct --port 30000. Log wording varies by release; the fields named here are long-standing.

Lab 1: watch the radix cache work. Send a request with a ~2,000-token prompt and note the prefill log line's #cached-token: 0. Send the identical prompt again and watch #cached-token jump to nearly the full prompt length, with time-to-first-token collapsing. Then curl -X POST localhost:30000/flush_cache and repeat to see it go cold. Concept taught: match, insert, and the fact that finishing a request publishes its KV.

Lab 2: break the cache with prompt ordering. Run 20 requests whose shared 1,500-token context comes first and sum the #cached-token values; then run 20 with a unique timestamp prepended and watch reuse drop to zero. Concept taught: radix sharing is prefix sharing, byte-for-byte from token one.

Lab 3: measure jump-forward. Ask for a ~40-field flat JSON object twice: once free-form ("respond in JSON"), once with a strict schema via response_format. Compare wall-clock generation time and the decode throughput log lines: the constrained run finishes in far fewer forward passes because keys and punctuation are jumped. Concept taught: the grammar fast path. (First schema use pays a one-time compile; send it twice and time the second.)

Lab 4: cache-aware scheduling. Prepare 64 requests, 8 groups of 8 sharing a distinct long prefix, and fire them all at once, interleaved across groups. Run once with the default policy and once with --schedule-policy lpm, comparing total completion time and summed #cached-token. Concept taught: the scheduler can raise hit rates by reordering admission.

Lab 5: pressure and retraction. Restart with --max-total-tokens 20000 (a debugging flag that caps the KV pool) and fire 64 concurrent requests each generating 1,000 tokens. Watch token usage climb toward 1.0 and retraction warnings appear as the scheduler pushes running requests back to the queue, yet every request completes. Concept taught: eviction and retraction as the pressure valves.

Lab 6: fork in the DSL. Run the compare program from Part V against the server and watch the logs: one prefill for the shared prompt, then the branches show up with high #cached-token, decoding concurrently. Concept taught: language structure mapping directly onto tree structure.

Part VIII: Understanding checks

What is SGLang in one sentence? A serving runtime co-designed with a programming interface, organized around reuse: a radix tree indexes the whole KV cache for prefix sharing, grammars skip tokens the format has determined, and the frontend language exposes program structure the runtime exploits.

What exactly does RadixAttention cache? Nothing beyond the ordinary KV pool: it is an index. The radix tree maps token-sequence prefixes to KV page indices in the shared pool, so "cached" prefixes are just finished computations left resident and evictable, discoverable by tree walk.

Why a radix tree instead of a hash map of prompts? A hash map can only find exact or precomputed-boundary matches; a radix tree finds the longest shared prefix of any new prompt against all history in one walk, handles mid-edge divergence by splitting, and gives eviction a natural leaf-LRU structure.

How does this differ from vLLM's prefix caching? Mechanism and emphasis. vLLM hashes fixed-size blocks into a map, sharing at block granularity as one feature of its paged allocator; SGLang indexes token-granularity pages (page size defaults to 1) in a tree that is the organizing structure of memory, feeding a cache-aware scheduler. The two systems have otherwise converged substantially.

Walk the process pipeline of SRT. The HTTP process holds the TokenizerManager, which tokenizes and forwards over ZMQ to the scheduler process owning the GPU; output IDs flow to the detokenizer process, then back to the TokenizerManager for the HTTP response. String work never blocks forward passes.

What happens when the KV pool fills? First, eviction: LRU leaves of the radix tree are freed, never nodes locked by running requests. If running requests still cannot grow, the scheduler retracts some back to the waiting queue to resume later. Frequent retraction means the pool is undersized for the concurrency, tunable via schedule conservativeness or memory fraction.

Why does constrained decoding need a bitmask rather than post-hoc validation? Sampling is committal: an illegal token, once emitted, has already consumed a step and poisoned the sequence. Masking logits before sampling makes invalid output unrepresentable, which is a guarantee, where retry-until-parses is a probability.

What makes jump-forward correct despite tokenizers? Jumped text is appended as a string and the boundary re-tokenized, because the concatenation of the last token and the jumped string may tokenize differently than the pieces; skipping that step would desynchronize the token sequence, the KV cache, and the text.

What does XGrammar add over a regex FSM? Context-free grammars via a pushdown automaton, so recursive structures like nested JSON are first-class; precomputed bitmasks for context-independent states with the context-dependent remainder computed at runtime, overlapped with the GPU; and its own jump-forward strings. It is SGLang's default backend.

Does a JSON schema make outputs correct? No, only parseable. Masking renormalizes probability among legal tokens, so the model can still assert a wrong value in flawless syntax, and the constrained distribution differs subtly from what unconstrained decoding plus rejection would give.

What do gen, select, and fork map to in the runtime? gen is an async generation slot filling a named variable; select scores each fixed option's continuation probability over the cached prefix instead of free-decoding; fork creates parallel branches whose shared parent prompt is one radix-tree prefix, so N branches cost one prefill.

When would you choose vLLM instead? When you need the broadest model, hardware, and quantization ecosystem, or your workload has little prefix overlap and no output constraints, where SGLang's signature machinery buys little. For serious deployments, benchmark both on your own traffic; they have converged enough that workload shape decides.

Throughput is fine but cache hit rate is near zero in production. What do you check? The load balancer first: round-robin across replicas defeats per-replica radix trees, so check for sticky or prefix-aware routing. Then prompt construction: dynamic content (timestamps, request IDs, shuffled few-shot examples) ahead of the static prefix splits the tree at the first divergent token.

Why can the scheduler overlap CPU work with the GPU? Because batch N+1's composition does not depend on batch N's sampled values, only on request states the scheduler already tracks, so it can build the next batch while the GPU runs the current one and reconcile afterwards; this overlap mode is the default event loop.

Part IX: Design lessons

Turn a cache into an index and it becomes an architecture. Prefix caching as a lookup table is a feature; a radix tree over all history makes reuse visible to the scheduler, the router, and the eviction policy, and suddenly the whole system can plan around it. The same promotion happens when databases turn a buffer cache into a buffer manager the planner can reason about.

Co-design the interface with the engine. fork, select, and gen exist because the runtime can honor them specially, and the runtime's radix tree exists because programs shaped like fork occur. API and engine designed together each make the other better; the same story as SQL and query optimizers, or map-reduce and its schedulers.

Spend compute only where there is a decision. Jump-forward is a profound pattern wearing a small trick's clothes: identify the parts of the work whose outcome is already determined and skip the expensive machine for them. Branch prediction, memoization, and incremental compilation are the same lesson in other clothes.

Pipeline by process, not by thread. Tokenizer, scheduler, and detokenizer as separate processes joined by queues buys isolation from Python's GIL and from each other's latency spikes, at the cost of serialization. Every serious serving system rediscovers this shape; so did print spoolers.

Keep the escape hatch simple. chunk_cache.py, the dumb no-sharing cache behind --disable-radix-cache, is small and boring on purpose: when the clever structure is suspected of a bug or mismatched to a workload, the fallback isolates it in one flag. Clever systems age well when their off switch is trustworthy.

Part X: Memorization framework

One sentence: SGLang bets that inference workloads are full of structure, repeated prefixes, constrained formats, branching programs, and builds the runtime so every one of those structures converts into skipped work.

HTTP → Tokenize → Grammar → Radix match → Batch → Forward → Mask/Jump → Sample → Detok → SSE → Insert into tree

The chain mapped to files (verified at v0.5.15, under python/sglang/srt/):

HTTP          entrypoints/http_server.py (+ entrypoints/openai/)
Tokenize      managers/tokenizer_manager.py     (HTTP process)
Grammar       constrained/xgrammar_backend.py    (compiled, cached)
Radix match   mem_cache/radix_cache.py           (index over the pool)
Batch         managers/scheduler.py, schedule_policy.py, schedule_batch.py
Forward       managers/tp_worker.py → model_executor/model_runner.py
                → layers/attention/* (+ sgl-kernel/)
Mask/Sample   layers/sampler.py + grammar bitmask; jump-forward skips passes
Detok         managers/detokenizer_manager.py    (own process)
Insert        radix tree keeps the finished KV, LRU-evictable

Memorize these:

The cache fact: the radix tree is an index, not a store; match on admission, insert on completion, evict LRU leaves, lock_ref pins what is running; page size defaults to one token, so sharing is token-precise.

The grammar fact: compile once and cache, mask logits every step, and when the automaton has a single outgoing path, append the string, re-tokenize the seam, and skip the forward passes entirely.

The pipeline fact: three processes, tokenizer to scheduler to detokenizer, joined by ZMQ; the scheduler overlaps building batch N+1 with executing batch N.

The deployment fact: the cache lives per replica, so routing decides the hit rate; prefix-aware or sticky routing above, static-prefix-first prompts below.

For how these serving-layer choices compose into a whole platform, my LLM serving design write-up covers the same ground from the requirements side, and the kernel-level story of attention over cached memory is on the FlashAttention page with the underlying math on the softmax page.

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Zheng et al., SGLang, Efficient Execution of Structured Language Model Programs, NeurIPS 2024. The paper this repository implements, and the source of RadixAttention, the compressed finite state machine behind jump-forward decoding, and the frontend language.
2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The paged KV allocator that SGLang shares with its closest peer, covered in the vLLM walkthrough.
3. Yu et al., Orca, A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. The iteration-level continuous batching that every modern serving engine, SGLang included, builds on.
4. Dong et al., XGrammar, Flexible and Efficient Structured Generation Engine for Large Language Models, 2024. The default grammar backend, the pushdown automaton with precomputed bitmasks that Part V describes.
5. Willard and Louf, Efficient Guided Generation for Large Language Models, 2023. The finite state machine formulation of constrained decoding behind the Outlines backend.
6. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The kernel idea behind the attention backends, covered in the FlashAttention walkthrough with the underlying math on the attention page.
7. Ye et al., FlashInfer, Efficient and Customizable Attention Engine for LLM Inference Serving, 2025. The other headline attention backend, designed for the paged and prefix-shared KV layouts this chapter is about.
8. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022. The draft-and-verify decoding family that SGLang ships in production form, one of the frontier areas Part VI defers.