vLLM

Part I: The mental model

HTTP request (OpenAI JSON)
      │
      ▼
API server ──► chat template ──► token IDs        vllm/entrypoints/openai/
      │
      ▼
AsyncLLM ──► Processor (validate, tokenize)       vllm/v1/engine/
      │            (IPC to a separate process)
      ▼
EngineCore busy loop                              vllm/v1/engine/core.py
      │
      ├─► Scheduler: who runs this step?          vllm/v1/core/sched/
      ├─► KVCacheManager: which blocks?           vllm/v1/core/
      ▼
GPU model runner ──► attention backend kernel     vllm/v1/worker/, vllm/v1/attention/
      │
      ▼
Sampler: logits ──► one token per sequence        vllm/v1/sample/
      │
      ▼
OutputProcessor ──► incremental detokenize        vllm/v1/engine/
      │
      ▼
SSE chunk back to the client

The one-sentence identity: vLLM is a small operating system for the KV cache, wrapped in an iteration-level scheduler, wrapped in an OpenAI-compatible server. Everything in the diagram above exists to keep one resource, GPU memory holding attention keys and values, as full of useful work as possible.

Two ideas carry the design. The first, PagedAttention, is about space: instead of allocating each request's KV cache as one contiguous region sized for the worst case, the cache is carved into fixed-size blocks and a per-request block table maps logical token positions to physical blocks, exactly like virtual memory. Fragmentation collapses to a fraction of one block per request, and sharing cached prefixes between requests becomes a pointer operation instead of a copy. The second idea, continuous batching, is about time: the batch is rebuilt on every single forward pass, so finished sequences leave immediately and waiting requests join immediately, rather than the batch draining to the slowest member. Paging is what makes that constant churn cheap.

One honest historical note before reading any code. vLLM shipped a ground-up engine rewrite, called V1, which became the default during 2025; the old V0 engine has since been removed. V1 moved the core loop into its own process, merged prefill and decode into a single token-budget scheduler (so chunked prefill is not a feature but the only mode), made prefix caching on by default, and dropped some V0 mechanisms, notably swap-based preemption and copy-on-write forking, in favor of simpler recompute plus prefix reuse. A great deal of writing about vLLM on the internet describes V0. This chapter describes V1, and flags the places where the famous explanation is the old one.

Part II: Using it

Installing

On Linux with an NVIDIA GPU, which is the first-class platform, installation is one command into a Python 3.10+ environment (the wheels bundle CUDA-enabled PyTorch, so a matching driver is the only system requirement):

pip install vllm
# or, faster and what the docs now recommend:
uv pip install vllm

AMD ROCm and CPU targets are supported through separate build paths described in the install docs. On macOS there is no GPU path; Apple silicon runs only the CPU backend, built from source (pip install -e . inside a clone selects the CPU device automatically on a Mac). That build is useful for poking at the API and the code under a debugger, not for real serving, and for the labs below you want a Linux GPU box.

First real session

vLLM has two front doors. The online one is an OpenAI-compatible server. Start it with a small model so the first run is quick:

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096

The startup logs walk through the whole boot sequence: config resolution, weight loading, a profiling pass that measures free VRAM, KV cache sizing ("GPU KV cache size: N tokens"), and CUDA graph capture. When you see the Uvicorn line reporting http://0.0.0.0:8000, the server is up. Talk to it with curl:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

You get back a standard chat completion object with choices[0].message.content and a usage block, which is why swapping a proprietary API for a self-hosted model is usually a one-line base URL change in the client. The offline front door is a Python class, good for batch jobs and evaluation sweeps:

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", max_model_len=4096)
params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain KV caching in one paragraph."], params)
print(outputs[0].outputs[0].text)

Going deeper: streaming, sampling, structure

Streaming is the OpenAI client's normal streaming, token deltas over server-sent events:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Count to ten slowly."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Sampling controls travel in the request: temperature, top_p, max_tokens, logprobs, seed, plus vLLM extensions via extra_body such as top_k. Structured output is exposed through the OpenAI structured-outputs surface, and vLLM enforces it server-side with a grammar backend (vllm/v1/structured_output/ holds the XGrammar, llguidance, and outlines integrations):

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Give me a user named Ada, age 36."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "user",
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
                "required": ["name", "age"],
            },
        },
    },
)

The mistakes beginners make

The most common one is benchmarking the engine with a sequential loop, which measures single-stream latency and concludes vLLM is slow. Continuous batching only pays off under concurrency:

# Wrong: this can never exercise batching.
for prompt in prompts:
    client.chat.completions.create(model=M, messages=msg(prompt))

# Right: give the scheduler something to schedule.
import asyncio
from openai import AsyncOpenAI
aclient = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="unused")

async def main():
    await asyncio.gather(*[
        aclient.chat.completions.create(model=M, messages=msg(p))
        for p in prompts
    ])
asyncio.run(main())

The second is sending raw text for an instruct model through /v1/completions and getting rambling output: the chat template never got applied, so the model saw no turn structure. Use /v1/chat/completions with messages unless you genuinely want raw continuation. The third is out-of-memory at startup on a model that "should fit": by default vLLM claims 92% of VRAM (--gpu-memory-utilization, default 0.92 as of v0.25.1) and then reserves KV cache for --max-model-len worth of context taken from the model config, which for modern 128k-context checkpoints is enormous. Capping --max-model-len to what your workload actually uses is the fix, and it is the single most useful flag to know. The fourth is treating --tensor-parallel-size as a speed knob: it exists to fit models that do not fit on one GPU, and for a model that already fits, running independent replicas usually beats splitting it.

Part III: When it is the right tool

vLLM is the right default when you are serving an open-weights transformer on server-class GPUs and you care about throughput under concurrent traffic: product APIs, internal model gateways, batch synthetic-data generation, evaluation sweeps, RLHF rollout workers. Its ecosystem breadth is a real feature: day-one support for most new open models, quantization formats, LoRA serving, and an API surface the whole tooling world already speaks.

The honest alternatives: SGLang overlaps almost completely and tends to shine on prefix-heavy and structured-output-heavy workloads, where its radix-tree cache and grammar fast path are the organizing principles rather than features; TensorRT-LLM trades flexibility for peak performance on NVIDIA hardware with ahead-of-time engine builds; llama.cpp and Ollama own the CPU, laptop, and single-user niche, where a big batch scheduler is pure overhead. If your traffic is one user on a MacBook, vLLM is the wrong shape.

The architecture-shaped warning: vLLM assumes it owns the GPU. It profiles free VRAM at startup and claims a fixed fraction of the whole device for weights plus KV cache. Co-locating two engines, or an engine plus a training job, on one GPU with default settings is the "writable SQLite file on NFS" of this system: it may boot, and then one process OOMs the other at peak load, or the KV caches get sized so small that both engines thrash through preemption.

Safe:                          Dangerous:
GPU 0 ── vllm A (0.92)         GPU 0 ── vllm A (0.92)
GPU 1 ── vllm B (0.92)                └─ vllm B (0.92)  ← both profiled
                                          the same free memory at boot;
                                          OOM or preemption thrash later

If you must share a device, lower --gpu-memory-utilization on both sides so the claims sum below 1.0, and accept that both engines will preempt more. The multi-replica version of the same warning appears in the SGLang chapter: a cache-blind load balancer in front of replicas quietly destroys prefix-cache hit rates.

Part IV: The full life of one request

The canonical operation is one streaming chat completion. Follow it end to end and you touch every load-bearing module in the repository. Paths below are the V1 engine tree as of v0.25.1.

Stage 1: HTTP arrival and the chat template

vllm serve boots a FastAPI app defined in vllm/entrypoints/openai/api_server.py; the chat route and its request/response models live in vllm/entrypoints/openai/chat_completion/ (older releases called this file serving_chat.py). The serving layer validates the JSON against the OpenAI protocol models, then renders the messages array through the model's chat template (a Jinja template shipped in the tokenizer config) into one flat prompt string with the model's role markers. This stage is deliberately thin: no scheduling decisions, no GPU state, just protocol translation.

Stage 2: AsyncLLM and the Processor

The serving layer hands the prompt to AsyncLLM (vllm/v1/engine/async_llm.py), the asyncio-friendly engine facade that both the server and the offline LLM class (vllm/entrypoints/llm.py) sit on. Its input processor (vllm/v1/engine/input_processor.py) tokenizes the prompt with the Hugging Face tokenizer, validates lengths against max_model_len, resolves sampling parameters into an engine request, and registers the request so output can be routed back to the right coroutine later.

Stage 3: Crossing into EngineCore

Here V1 differs visibly from V0: the core engine runs in its own process. vllm/v1/engine/core_client.py serializes the new request over a message-queue IPC channel to EngineCore (vllm/v1/engine/core.py), which runs a tight busy loop: pull inputs, call step(), push outputs. The separation keeps Python HTTP handling, tokenization, and detokenization off the process that must launch GPU work every few milliseconds, so an expensive JSON parse in the API layer cannot stall a decode step.

Stage 4: The scheduler decides the step

Each iteration, EngineCore asks the scheduler (vllm/v1/core/sched/scheduler.py) what to run. The V1 scheduler has no prefill/decode dichotomy: every request simply has num_computed_tokens and a total it needs, and the scheduler hands out tokens from a per-step budget (max_num_batched_tokens) across running requests first, then admits waiting ones. Our new request gets admitted; if its prompt is long, it receives only a chunk of the budget this step and continues next step, which is chunked prefill happening not as a feature but as the natural consequence of the design. Decoding requests in the same step each get one token of budget, so prefill and decode mix freely in a single batch.

Stage 5: Paging the KV cache

Before scheduling tokens, the scheduler asks the KVCacheManager (vllm/v1/core/kv_cache_manager.py) for memory. The manager first checks the prefix cache: prompt blocks are hashed by content, and any full block whose hash chain matches an already cached block is reused rather than recomputed, which is why a shared system prompt costs its prefill once per server, not once per request. Fresh blocks come from the BlockPool (vllm/v1/core/block_pool.py), and the request's block table, the logical-to-physical map, grows accordingly. If the pool is empty, this is where preemption happens (Part V).

Stage 6: The forward pass

The scheduler's output goes to the model executor and lands in the GPU worker (vllm/v1/worker/gpu_worker.py) and model runner (vllm/v1/worker/gpu_model_runner.py). The runner maintains persistent GPU-side input buffers and block tables (vllm/v1/worker/block_table.py) and applies the scheduler's diff to them, rather than rebuilding tensors from scratch each step; decode steps replay pre-captured CUDA graphs to erase Python launch overhead. The model itself, in vllm/model_executor/models/, is deliberately plain PyTorch; all the cleverness is injected at the attention layer, which dispatches to a backend (vllm/v1/attention/backends/) that reads and writes the paged KV cache via the block table. For why that kernel can stream through a sequence it never holds in one piece, see the online-softmax derivation on the softmax page and the FlashAttention chapter.

Stage 7: Sampling

The forward pass produces one logits row per scheduled sequence. The sampler (vllm/v1/sample/sampler.py) applies penalties, temperature, top-k/top-p, and any structured-output bitmask, then samples one token ID per sequence on the GPU; speculative decoding routes through the rejection sampler (vllm/v1/sample/rejection_sampler.py) instead. The chosen token is appended to the request's state, and its KV entry was already written into the request's current block during the forward pass.

Stage 8: Detokenization and the stream back

Token IDs return over IPC to the API process, where the OutputProcessor (vllm/v1/engine/output_processor.py) and the incremental detokenizer (vllm/v1/engine/detokenizer.py) turn them into text. Detokenization is stateful on purpose: BPE pieces do not map one-to-one to characters, so the detokenizer buffers until it can emit clean UTF-8, checks stop strings, and yields a delta. AsyncLLM routes the delta to the waiting coroutine, and the serving layer wraps it in an SSE chat.completion.chunk. This loop, stages 4 through 8, repeats every iteration until EOS, a stop string, or max_tokens.

Stage 9: Finish and free

On completion the scheduler frees the request's blocks back to the pool. Freed does not mean erased: blocks keep their content hashes and sit in the free list still indexed by the prefix cache, so a follow-up turn of the same conversation can reclaim them. They are only actually recycled when allocation pressure evicts them, LRU from the free queue.

Part V: Internals deep dives

PagedAttention and the KV block manager

The memory problem it solves is worth restating with the paper's numbers: pre-vLLM engines allocated each request's KV cache contiguously at maximum length, and the SOSP paper measured that existing systems wasted most of their KV memory to fragmentation and over-reservation, directly capping batch size and therefore throughput. The fix is virtual memory: fixed-size blocks (16 tokens is the classic CUDA default; the exact value is resolved per platform and attention backend in current releases), a block table per request, and an attention kernel that follows the table.

Request A: "The cat sat on the mat and then ..."
logical blocks:   [0] [1] [2]
block table A:    0→7  1→2  2→9

Request B: same system prompt, different question
block table B:    0→7  1→2  2→4     ← blocks 7,2 shared, ref_cnt=2

Physical pool:  blk2 blk4 blk7 blk9 ...   free queue: 5,8,1,...

Sharing is reference counting. In the original paper and the V0 engine, parallel sampling and beam search forked sequences that shared blocks copy-on-write: a shared block was duplicated only when one branch tried to append into it. The V1 engine dropped explicit fork/copy-on-write; parallel sampling (n>1) is handled as sibling requests (vllm/v1/engine/parallel_sampling.py) whose common prompt is shared through the prefix cache instead, which is the same memory win with far simpler bookkeeping. If you read the paper's copy-on-write section and then look for it in V1, this is why you cannot find it.

Prefix caching is the block manager's second job. Every full block gets a hash of its tokens chained with its prefix's hash, stored in a map from hash to physical block. Because the hash covers the entire prefix, a hit at block k guarantees the whole prefix matches, and lookup is per-block, so a 10,000-token shared prefix is recovered in a few hundred hash probes. It is on by default (enable_prefix_caching defaults to true in V1) precisely because it is nearly free: cached blocks live in the free list, so an idle cache costs no capacity. The famous misconception to correct: prefix caching does not "use extra memory"; it indexes memory that would otherwise sit free, and eviction reclaims it the moment allocation needs it. The real trade-offs are the hashing overhead on very low-reuse workloads, and block granularity: only whole matched blocks are reused, so a shared prefix's tail partial block is recomputed.

Continuous batching and the scheduler

Static batching runs a batch until every member finishes, so one long generation holds hostages. Continuous batching, inherited from the Orca paper, reschedules every iteration. The V1 scheduler implements it with one mechanism, the token budget:

step N   budget = 8192 tokens
  running decodes: 300 reqs × 1 token          =  300
  running chunked prefill: 1 req               = 4096 (its next chunk)
  admit from waiting: new req, 3796-token chunk = 3796
  ──────────────────────────────────────────────  8192  → launch one batch

The knobs are exactly the quantities in the diagram: --max-num-batched-tokens is the budget, --max-num-seqs caps concurrent requests, and --long-prefill-token-threshold tunes how aggressively long prompts are chunked. Chunked prefill exists because a monolithic 100k-token prefill would occupy the GPU for whole seconds, stalling every decoding request's next token; slicing it keeps inter-token latency flat for everyone else at a small cost in prefill completion time. In V1 this is always on, one scheduling path for everything.

Preemption is the pressure valve. When the block pool cannot supply a running request's next block, the scheduler preempts the lowest-priority (in FCFS, the most recently arrived) running request: _preempt_request in vllm/v1/core/sched/scheduler.py frees its blocks, resets num_computed_tokens to zero, and requeues it. V1 preemption is recompute-only; the V0 option of swapping KV blocks to CPU RAM is gone from the scheduler, on the argument that recompute is simpler, prefix caching often recovers much of the work anyway, and KV offload to other tiers is better handled by the separate connector/offload machinery (vllm/v1/kv_offload/). Preemption shows up in logs and in the vllm:num_preemptions counter; if you see it steadily under normal load, your KV cache is undersized: lower --max-model-len, raise --gpu-memory-utilization, or shrink --max-num-seqs.

The misconception to correct here: continuous batching does not mean your request shares a batch with others "at the same position". Sequences in one forward pass are at wildly different lengths and phases, which is precisely why attention must be computed per-sequence against per-sequence block tables, and why the naive picture of a rectangular batch tensor stops applying the moment you understand this engine.

The attention backend dispatch layer

vLLM does not have one attention kernel; it has a dispatch layer. vllm/v1/attention/backends/ contains, among others, flash_attn.py (FlashAttention with paged KV, the default on modern NVIDIA GPUs), flashinfer.py, triton_attn.py (the portable Triton fallback), ROCm backends, and an mla/ family for DeepSeek-style multi-head latent attention; registry.py maps names to implementations and selection weighs platform, model architecture, head size, and dtype, with the VLLM_ATTENTION_BACKEND environment variable as the manual override. The interface each backend implements is narrow: build per-step metadata from the scheduler's output, then run attention given query tensors, the paged KV pool, and block tables. That narrowness is what let the project ride kernel progress for three years without touching the scheduler.

A historical note that trips up repo readers: the original hand-written paged_attention_v1/v2 CUDA kernels from the paper era no longer dominate; in the current tree csrc/ is mostly cache ops, quantization, MoE, and fused kernels, while paged attention itself typically executes inside FlashAttention or FlashInfer, both of which grew native paged-KV support. The idea outlived its original kernel, which is the healthiest possible outcome. The kernel math these backends share, tiling plus online softmax, is derived on the FlashAttention page.

Part VI: Reading the repository

The repo is big (thousands of files), but one request's life touches everything worth reading first. A staged plan, verified against v0.25.1:

Stage 0, orientation (one evening). Read the PagedAttention paper (Kwon et al., SOSP 2023) and skim the V1 blog post in the project docs. Questions you should be able to answer: why does KV memory cap batch size? What are internal and external fragmentation in this context? What did V1 change and why?

Stage 1, the front doors. Read vllm/entrypoints/llm.py, then vllm/entrypoints/openai/api_server.py and vllm/entrypoints/openai/chat_completion/. Questions: where does the chat template get applied? What does the serving layer know about the GPU (answer: nothing)?

Stage 2, the engine spine. Read vllm/v1/engine/async_llm.py, input_processor.py, core_client.py, core.py, then output_processor.py and detokenizer.py. Questions: which process does tokenization run in? What exactly crosses the IPC boundary in each direction? Why is detokenization stateful?

Stage 3, the brain. Read vllm/v1/core/sched/scheduler.py top to bottom, then vllm/v1/core/kv_cache_manager.py, block_pool.py, and kv_cache_utils.py (the hashing lives there). Questions: what is the token budget and who spends it? Walk through a preemption. How does a prefix cache hit change what the scheduler asks the runner to compute?

Stage 4, the muscle. Read vllm/v1/worker/gpu_model_runner.py (it is long; read for structure), block_table.py, one model file such as vllm/model_executor/models/llama.py, and one backend, vllm/v1/attention/backends/flash_attn.py. Questions: what state persists on the GPU across steps? Where does the block table actually get consumed? Why are model files so plain?

Where not to start: csrc/, the quantization layers, speculative decoding, and the distributed executor. All are deep specialties that make sense only once the single-GPU request path is solid, and none of them will teach you what vLLM is.

Part VII: Hands-on labs

All labs assume a Linux GPU box with vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096 running unless stated. Exact log wording and metric names shift between releases; the quantities do not.

Lab 1: watch continuous batching happen. Fire 50 concurrent requests with the async client from Part II while watching the periodic engine log line, which reports running and waiting request counts and KV cache usage. Observe requests joining and leaving between iterations, and confirm with curl -s localhost:8000/metrics | grep num_requests. Concept taught: iteration-level scheduling.

Lab 2: measure prefix caching. Send a request with a 2,000-token system prompt, then send it again with a different final question. Compare the two time-to-first-token values, and check curl -s localhost:8000/metrics | grep prefix_cache (V1 exposes query and hit counters). Then restart with --no-enable-prefix-caching and repeat. Concept taught: block hashing and reuse.

Lab 3: force preemption. Restart with --gpu-memory-utilization 0.35 --max-model-len 4096 and send 100 concurrent requests each asking for max_tokens=1500 of output. Watch for preemption warnings and the num_preemptions counter, and note that every preempted request still completes correctly. Concept taught: recompute preemption as the pressure valve.

Lab 4: chunked prefill and interference. Restart with --max-num-batched-tokens 1024. Stream a short chat while a second client sends a maximally long prompt, and watch the streaming cadence of the short chat stay smooth; then raise the budget to 8192 and watch the long prefill visibly stutter the stream. Concept taught: the token budget trades prefill throughput against decode latency.

Lab 5: swap attention backends. Run VLLM_ATTENTION_BACKEND=TRITON_ATTN vllm serve ..., then benchmark both backends with the bundled harness (recent releases ship vllm bench serve, plus latency and throughput subcommands) and compare. Setting a nonsense backend name is a legitimate trick: the error lists the valid names for your version. Concept taught: the dispatch layer is a real seam you can operate.

Lab 6: one step under a debugger. Run the offline LLM example under pdb with VLLM_ENABLE_V1_MULTIPROCESSING=0 so the engine stays in-process, break inside the scheduler's schedule(), and inspect a request's block table and num_computed_tokens across three iterations. Concept taught: everything in Part IV, made concrete.

Part VIII: Understanding checks

What problem does PagedAttention actually solve? KV cache memory waste. Contiguous per-request allocation at maximum length wastes most of KV memory to internal and external fragmentation, and since KV memory caps batch size, wasted memory is wasted throughput. Paging bounds waste to a fraction of one block per request and makes sharing a pointer operation.

Why does batch size control throughput in LLM serving? Decode is memory-bandwidth-bound: each step streams all weights from HBM regardless of batch size, so tokens per second scales almost linearly with batch until compute or memory runs out. More resident sequences amortize the same weight traffic over more tokens.

What is the difference between continuous and static batching? Static batching admits and releases requests only at batch boundaries, so the batch drains to its slowest member. Continuous batching rebuilds membership every forward pass: finished sequences exit and waiting ones join at iteration granularity.

What is a block table? The per-request map from logical block index to physical block ID in the KV pool, the exact analogue of a page table. The attention kernel follows it to gather a sequence's keys and values from non-contiguous memory.

How does prefix caching find a reusable prefix? Each full block is hashed over its own tokens chained with its prefix block's hash, and the hash maps to a physical block. The chaining means a hit at block k certifies the entire prefix, so matching is a per-block lookup, not a tree search.

What happens when the block pool runs dry mid-generation? The V1 scheduler preempts a running request: frees its blocks, zeroes its computed-token count, and requeues it for later recomputation, which prefix caching often makes partial rather than total. V0 additionally offered swapping blocks to CPU; V1 removed that mode from the scheduler.

Why chunk long prefills? A monolithic long prefill monopolizes the GPU for many milliseconds to seconds, freezing inter-token latency for every decoding request. Chunking slices the prompt across steps so each batch mixes a prefill chunk with everyone's decodes, trading a little prefill completion time for flat decode latency.

Why is detokenization incremental and stateful? BPE token boundaries do not align with character boundaries, so a token can be an unfinished UTF-8 fragment or the middle of a stop string. The detokenizer buffers per request and only emits deltas it knows are final text.

What did the V1 rewrite change architecturally? The engine core moved to its own process behind an IPC boundary, the scheduler collapsed prefill and decode into one token-budget mechanism with chunked prefill always on, prefix caching became the default, and swap preemption and copy-on-write forking were dropped in favor of recompute plus prefix reuse.

Where did copy-on-write go? It existed in V0 for beam search and parallel sampling forks. V1 implements n>1 as sibling requests sharing their prompt through the prefix cache, achieving the sharing without fork bookkeeping in the block manager.

When would you pick SGLang or TensorRT-LLM instead? SGLang for workloads dominated by shared prefixes or grammar- constrained output, where its radix cache and jump-forward decoding are structural advantages; TensorRT-LLM when you can pay an ahead-of-time build per model and GPU for peak NVIDIA performance. For single-user local inference, llama.cpp-family tools beat both.

A model "fits in VRAM" but vLLM OOMs at startup. Why? vLLM pre-reserves KV cache sized by gpu-memory-utilization and max-model-len on top of weights, and profiles at boot. A 128k default context can demand tens of gigabytes of KV alone; capping --max-model-len or lowering utilization fixes it.

Your p99 latency spikes but average throughput looks fine. What do you check first? Preemption counters and KV cache usage in /metrics: preemption recompute shows up exactly as tail latency. Second, long-prompt interference: if max-num-batched-tokens is large, one giant prefill chunk can stall a step for everyone.

Does the attention kernel pay for paging? Yes, a small indirection cost: it gathers KV through the block table instead of striding contiguous memory. The design accepts single- digit-percent kernel overhead to multiply achievable batch size, which is a spectacular trade at the system level.

Part IX: Design lessons

Old ideas port to new resources. vLLM's core insight is literally paged virtual memory applied to a new scarce resource. The same move recurs everywhere: databases paging buffers, GC generations, network buffer pools. When a new resource is scarce and fragmented, check whether the OS already solved it in 1970.

Separate the control plane from the data plane. The scheduler decides in Python at microsecond scale what the kernels execute at millisecond scale, connected by a narrow description (block tables, token budgets). Routers versus forwarding ASICs, query planners versus executors, Kubernetes versus kubelets: same shape, and it is why vLLM can evolve policy without touching CUDA.

Make the fast path the only path. V0 accumulated modes: chunked prefill optional, prefix caching optional, two preemption strategies. V1's bet is that one always-on mechanism, tuned well, beats a matrix of flags, because every optional path is a path that rots untested. The same argument shows up in storage engines that delete their non-WAL modes.

Keep the extension surface narrow and boring. Hundreds of model architectures integrate as plain PyTorch because all the cleverness is concentrated behind the attention interface and the block manager. Cleverness that must be reimplemented per integration kills an ecosystem; vLLM's model zoo is large precisely because contributing a model is dull.

Rewrites are honest when they delete. V1 did not just refactor; it removed swap preemption, copy-on-write, and the prefill/decode split, accepting temporary feature regressions to get a simpler invariant set. The willingness to shrink the design is what distinguishes a real second system from second-system syndrome.

Part X: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. The vLLM paper, paging the KV cache like virtual memory so fragmentation stops capping batch size. The LLM serving platform write-up on this site derives the same design from requirements.
2. Yu et al., Orca, A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. Introduces iteration-level scheduling, the continuous batching that the V1 token-budget scheduler generalizes.
3. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The tiled IO-aware kernel behind the default attention backend, derived in the FlashAttention walkthrough.
4. Milakov and Gimelshein, Online normalizer calculation for softmax, 2018. The one-pass softmax trick that lets an attention kernel stream through a sequence, worked through on the softmax page.
5. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI 2024. The chunked-prefill and stall-free scheduling argument that V1 turned into its only scheduling mode.
6. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022. The draft-and-verify sampling that vLLM's rejection sampler implements.
7. Ye et al., FlashInfer, Efficient and Customizable Attention Engine for LLM Inference Serving, MLSys 2025. The other headline backend in the dispatch layer, attention kernels built natively around paged and block-sparse KV.
8. Zheng et al., SGLang, Efficient Execution of Structured Language Model Programs, 2023. RadixAttention and the grammar fast path, the closest rival design, covered in the SGLang walkthrough.
9. DeepSeek-AI, DeepSeek-V2, A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024. Introduces multi-head latent attention, the reason the backend tree grew its mla/ family. The attention variants page covers the idea.
10. Dong et al., XGrammar, Flexible and Efficient Structured Generation Engine for Large Language Models, 2024. The grammar engine behind the default structured-output backend.

Part XI: Memorization framework

One sentence: vLLM pages the KV cache so memory is never wasted and rebuilds the batch every iteration so time is never wasted, and the rest of the repository is what those two ideas need to survive real models, real GPUs, and real traffic.

HTTP → Template → Tokenize → IPC → Schedule → Blocks → Forward → Sample → Detok → SSE

The chain mapped to files (V1, verified at v0.25.1):

HTTP/Template  entrypoints/openai/api_server.py, chat_completion/
Tokenize       v1/engine/input_processor.py  (in AsyncLLM's process)
IPC            v1/engine/core_client.py ⇄ core.py  (EngineCore process)
Schedule       v1/core/sched/scheduler.py    (token budget, preemption)
Blocks         v1/core/kv_cache_manager.py, block_pool.py
Forward        v1/worker/gpu_model_runner.py + v1/attention/backends/
Sample         v1/sample/sampler.py
Detok/SSE      v1/engine/output_processor.py, detokenizer.py

Memorize these:

The memory fact: KV blocks of ~16 tokens, block table per request, waste bounded by one partial block, sharing by refcount, prefix reuse by chained content hash, eviction LRU from the free queue.

The time fact: one token budget per step (max-num-batched-tokens) spent on running requests first; prefill is just a request that wants many tokens, decode one that wants one, so chunked prefill is free structurally.

The pressure fact: out of blocks means preempt latest, free, zero the computed count, requeue, recompute; V1 has no scheduler swap mode.

The version fact: V0 versus V1: separate core process, unified scheduler, prefix caching default on, CoW and swap gone. Most old blog posts describe V0.