TensorRT-LLM

Part I: The mental model

OpenAI-style HTTP request
   │
   ▼
trtllm-serve, FastAPI layer        (tensorrt_llm/serve/)      protocol, chat template
   │
   ▼
LLM API                            (tensorrt_llm/llmapi/)     config, tokenizer, lifecycle
   │
   ▼
GenerationExecutor proxy ⇄ worker  (tensorrt_llm/executor/)   IPC across processes
   │
   ▼
PyExecutor loop                    (tensorrt_llm/_torch/pyexecutor/)  the serving heartbeat
   │
   ├──► schedulers ────────────────► C++ batch_manager algorithms (via bindings)
   ├──► KV cache manager ──────────► C++ paged block pools
   ▼
model engine                       (_torch/models, modules)   PyTorch modules, CUDA graphs
   │
   ▼
kernels                            (cpp/tensorrt_llm/kernels, cutlass_extensions)
   │
   ▼
tensor cores: FP16 / FP8 / NVFP4   (Hopper, Blackwell)

One sentence of identity: TensorRT-LLM is a PyTorch-fronted, C++-cored serving engine whose entire reason to exist is extracting the last factor of performance from NVIDIA silicon, including the numeric formats that silicon was designed around. The diagram has an unusual shape compared with other engines: Python owns the definition of the model and the orchestration of the loop, while C++ owns the decisions made millions of times per second, scheduling, KV cache block management, and the kernels. The boundary between the two is crossed through generated bindings (the cpp/tensorrt_llm/nanobind/ tree), and learning where that boundary sits is most of learning the repository.

The historical shape matters too. For its first two years the project was a compiler: you translated a checkpoint into a TensorRT engine, a serialized, fused, kernel-selected execution plan, and the runtime replayed it. Since the 1.0 release the default is a PyTorch-native backend, where models are ordinary torch.nn.Module code calling hand-tuned kernels, and on current main the engine-building CLI no longer even ships. The compile-ahead-of-time philosophy did not die; it retreated into the kernels, the CUDA graphs, and the autotuner, which is exactly where it still pays. Part IV tells the live flow, then the legacy flow as the history that explains the architecture's scars.

Part II: Using it

Installing

This is Linux-plus-NVIDIA software with real prerequisites, which is why the container is the path of least pain. The prebuilt images on NGC come in a release flavor (ready to serve) and a devel flavor (for building from source):

docker pull nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc21

docker run --rm -it --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc21

The --ipc=host flag is not optional decoration; the docs call it out because the multi-process executor shares memory between processes and dies with a bus error without it. Bare metal installation works on a machine that already has the CUDA toolkit (13.1 for current releases), a matching PyTorch build, and OpenMPI development headers:

pip3 install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu130
sudo apt-get -y install libopenmpi-dev
pip3 install tensorrt_llm

Python 3.10 or newer is required; the docs test against Ubuntu 24.04 with Python 3.12. There is no macOS story and no non-NVIDIA story, by design. If those version pins already feel fragile to you, that instinct is correct and is the strongest argument for the container.

First session

The offline front door is the LLM API, which takes a Hugging Face model id and just runs it, no build step:

from tensorrt_llm import LLM, SamplingParams

llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
prompts = ["Hello, my name is", "The capital of France is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

for output in llm.generate(prompts, sampling_params):
    print(output.outputs[0].text)

First run downloads the checkpoint and spends a noticeable warmup period capturing CUDA graphs and tuning kernels; subsequent generations are fast. The online equivalent is an OpenAI-compatible server:

trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
       "messages": [{"role": "user", "content": "Where is New York?"}],
       "max_tokens": 32}'

Beside trtllm-serve the wheel installs exactly two other commands, trtllm-bench for standardized throughput and latency runs and trtllm-eval for accuracy checks. Configuration beyond flags goes in a YAML file passed as --extra_llm_api_options, whose keys mirror the LLM API arguments; this one file is where KV cache sizing, scheduler policy, and parallelism live:

# extra.yaml
kv_cache_config:
  free_gpu_memory_fraction: 0.85
  enable_block_reuse: true

trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --tp_size 2 --extra_llm_api_options extra.yaml

The mistakes beginners make

The most common one in 2026 is following a 2024 tutorial. Wrong and right:

# wrong (the old three-step engine workflow; trtllm-build
# no longer ships in current releases)
python convert_checkpoint.py --model_dir ... --output_dir ckpt/
trtllm-build --checkpoint_dir ckpt/ --output_dir engine/
trtllm-serve engine/

# right: the PyTorch backend loads the HF checkpoint directly
trtllm-serve "meta-llama/Llama-3.1-8B-Instruct"

Second, treating out-of-memory at startup as a model-size problem. The KV cache manager claims a large fraction of the GPU memory left after weights by design, so a healthy server runs near full memory; if startup fails, the knob is free_gpu_memory_fraction, not a smaller model. Third, benchmarking with one request at a time and concluding the engine is unimpressive: the entire architecture is built for concurrent load, and single-stream latency is the one metric where the heavyweight machinery shows the least. Use trtllm-bench with a realistic request rate before judging.

Part III: When it is the right tool

Choose TensorRT-LLM when the hardware is NVIDIA, the workload is production-scale, and you intend to use what the silicon vendor knows: FP8 on Hopper, NVFP4 on Blackwell, the fastest attention and GEMM kernels NVIDIA writes for its own chips, and features like disaggregated prefill/decode and wide parallelism that are developed against NVIDIA's own datacenter deployments. It is also the natural choice when you are already inside the NVIDIA stack, serving behind Triton Inference Server (the triton_backend/ tree ships in-repo) or consuming NVIDIA's pre-quantized checkpoints from Hugging Face.

The main alternative is vLLM, which runs on more hardware, has the larger open ecosystem, and is usually simpler to operate; for many teams its performance is close enough that operability wins. SGLang competes at the same performance tier with a strength in structured generation and multi-turn caching. llama.cpp is the answer when the deployment is a laptop rather than a fleet. The honest framing is that TensorRT-LLM buys peak performance and earliest access to new NVIDIA hardware features at the cost of a heavier dependency stack and a single-vendor commitment.

The architecture-shaped warning is inherited from the compiler era but still bites wherever engines linger: a TensorRT engine is not a model, it is a compiled artifact pinned to a GPU architecture and library version. Teams that stored engine files in a registry and deployed them across a mixed fleet learned that an engine built on H100 will not run on A100, and an upgrade of TensorRT could invalidate the whole shelf. The PyTorch backend dissolves most of this by compiling nothing ahead of time, but the same shape reappears wherever you cache tuned artifacts across heterogeneous machines:

safe:      checkpoint registry ──► per-SKU build/warmup at deploy ──► H100 fleet
                                            └────────────────────────► A100 fleet
dangerous: one engine/tuning artifact ──► copied across mixed SKUs ──► crashes,
           silent slow paths, or version-invalidated caches

Part IV: The full life of one request

The canonical operation: a chat completion POSTed to trtllm-serve, followed through the default PyTorch backend to the tensor cores and back. Every stage names real modules from the tree.

Stage 1: the command

trtllm-serve is a console script mapping to tensorrt_llm/commands/serve.py. It assembles LLM API arguments from flags plus the --extra_llm_api_options YAML, selects the backend (pytorch by default, with an experimental AutoDeploy variant), constructs the LLM object, and hands it to the OpenAI server in tensorrt_llm/serve/openai_server.py, a FastAPI application.

Stage 2: the OpenAI protocol layer

serve/openai_protocol.py defines the request models; the server validates your JSON, applies the model's chat template through the tokenizer to turn messages into one prompt string, and submits the prompt to the LLM API, getting back an async stream of results that it will re-encode as SSE chunks. This layer is deliberately thin and is the right place to read first, because everything below it is engine-agnostic: the same code path serves both backends.

Stage 3: the LLM API

tensorrt_llm/llmapi/llm.py is the stable public face (exported at the package root, so from tensorrt_llm import LLM), and llmapi/llm_args.py is the configuration monster where every knob in the system is declared as a typed field, KV cache config, scheduler config, parallel mapping, speculative decoding, quantization. Tokenization happens here (llmapi/tokenizer.py wraps the Hugging Face tokenizer), so what travels further down is token ids plus sampling parameters.

Stage 4: the executor boundary, and a process hop

The LLM API does not run the model in your process. The tensorrt_llm/executor/ package implements a GenerationExecutor split into a proxy (proxy.py) in the API process and one or more workers (worker.py, base_worker.py) holding the GPU, launched under MPI for multi-rank runs, with requests and results crossing over ZeroMQ-based IPC (ipc.py) and responses post-processed in dedicated processes (postproc_worker.py). The isolation is deliberate: the GIL-holding web frontend and detokenization work cannot stall the process that must keep the GPU fed. Your request becomes a GenerationRequest (request.py), is serialized across, and registers a GenerationResult (result.py) that will stream tokens back the other way.

Stage 5: the PyExecutor loop

Inside the worker lives the heartbeat: tensorrt_llm/_torch/pyexecutor/py_executor.py. New requests land in an executor_request_queue and become LlmRequest objects (llm_request.py), each carrying its lifecycle state (context phase, generation phase, finished). The loop body runs once per model step: fetch new requests, schedule, allocate KV blocks, run the model, sample, stream results, free finished requests. By default the engine runs the overlap variant, _executor_loop_overlap, which prepares and schedules step N+1 while the GPU still executes step N, hiding CPU bookkeeping behind GPU time.

Stage 6: scheduling, in C++ even here

Each iteration, the scheduler decides which requests run. pyexecutor/scheduler/scheduler.py defines the interfaces, but the default implementations, BindCapacityScheduler and BindMicroBatchScheduler, are thin wrappers over the C++ algorithms in cpp/tensorrt_llm/batch_manager/capacityScheduler.cpp and microBatchScheduler.cpp, reached through the nanobind bindings. The capacity scheduler answers "which requests may occupy the engine given KV memory," under a policy (GUARANTEED_NO_EVICT by default, MAX_UTILIZATION or STATIC_BATCH as alternatives); the micro-batch scheduler then packs the chosen requests' context chunks and generation steps into this iteration's batch under the token budget. This stage is the deep dive in Part V.

Stage 7: KV cache blocks

The resource_manager.py layer asks the KV cache manager, again C++ underneath (batch_manager/kvCacheManager.cpp), to allocate blocks for every scheduled request. The cache is paged, 32 tokens per block by default (KvCacheConfig.tokens_per_block), with reuse of blocks across requests that share prefixes and an eviction policy (evictionPolicy.cpp) governing the block pool. If your prompt shares a system prefix with earlier requests and block reuse is enabled, this is the stage where most of its "prefill" silently vanishes.

Stage 8: the model engine

pyexecutor/model_engine.py owns the forward pass. The model is genuine PyTorch: architecture classes in _torch/models/ composed from optimized modules in _torch/modules/, with attention dispatched through _torch/attention_backend/ to the TRT-LLM fused attention kernels (or FlashInfer), and custom ops registered in _torch/custom_ops/ so torch sees the hand-written kernels as ordinary operators. Two compile-era survivors do the performance work here: CUDA graphs (cuda_graph_runner.py) capture whole decode iterations at standard batch sizes so a step replays with almost no launch overhead, and the autotuner (_torch/autotuner.py) picks the best kernel variant per shape at warmup, which is why the first requests after startup are slower.

Stage 9: kernels and tensor cores

The bottom of the stack is cpp/tensorrt_llm/kernels/ plus cutlass_extensions/, with specialized trees vendored beside them (deep_gemm, flash_mla, deep_ep for expert parallelism). These are the CUTLASS-based GEMMs and fused attention kernels that know about FP8 and NVFP4 tensor-core datapaths, exported back to Python through the torch-op layer in cpp/tensorrt_llm/thop/. For one decode step of one request in a batch of hundreds, the GPU runs a handful of fused kernels per layer, most of them reading quantized weights the chip natively understands.

Stage 10: sampling and the way back

Logits return to pyexecutor/sampler/, which applies temperature, top-k/top-p and friends batch-wide (with guided_decoder.py constraining tokens to a grammar when structured output was requested; the softmax being sampled from is the same one derived on my softmax page). New token ids update each LlmRequest; finished requests release their KV blocks; responses stream through the IPC layer to the post-processing workers for detokenization, then to the FastAPI layer, which emits your SSE chunk. The loop, meanwhile, has already started the next iteration with a slightly different batch, which is the whole point of Part V's deep dive.

The legacy flow, as history

Until the 1.0 release (September 2025) the primary path was different in kind. You ran a per-model convert_checkpoint.py, then trtllm-build, which traced the model through TensorRT: layers fused, precisions fixed, one kernel tactic chosen per op by measuring candidates on your GPU, memory statically planned, everything serialized into engine files plus a config.json. The C++ runtime then loaded and replayed that plan. The payoff was real, work done once instead of per step; the cost was that every new architecture, every new attention trick, and every dynamic behavior had to be expressible in the compiler first, and in a field shipping novel models monthly that became the bottleneck. The 1.0 release made the PyTorch backend the default and stabilized the LLM API on top of it, and on current main the engine-building CLI is gone from the shipped entry points. Reading the C++ runtime/ tree today is reading well-organized history that still explains why the executor, batch manager, and kernel layers have the shapes they do.

Part V: Deep dive: ahead-of-time compilation, and what an engine was

The idea TensorRT-LLM inherited from TensorRT is that inference is a compiler problem: if the graph, shapes, precisions, and target chip are known ahead of time, you can make every expensive decision offline. Concretely an engine bakes in three classes of decision. Fusion: chains like GEMM + bias + activation, or an entire attention block, become single kernels, eliminating memory round-trips between ops. Tactic selection: for each fused op, TensorRT times competing kernel implementations on the actual GPU and records the winner. Static planning: activation memory is laid out once, so execution is a replay with no allocator in the loop.

AOT era:   checkpoint ─► trace ─► fuse ─► time tactics ─► serialize engine
           runtime: replay plan            (fast, rigid, per-GPU artifact)

JIT era:   checkpoint ─► PyTorch modules ─► custom fused kernels
           + autotune at warmup + CUDA graph capture per batch size
           (flexible, and the compile-time wins are re-earned piecemeal)

Why did the flexible side win? Because the rigid side taxed exactly the thing the field optimizes for: time-to-new-model. Each architecture needed compiler support before it could run at all; dynamic control flow (mixture-of-experts routing, speculative decoding trees) fought the static-graph worldview; and the artifact pinning from Part III made operations brittle. The revealing part is what the PyTorch backend kept. CUDA graph capture is a miniature engine: a recorded, replayable execution plan for the decode step, minus the portability problems. The autotuner is tactic selection moved to warmup. The fused kernels are the fusion pass, written by hand where it matters instead of derived for every op. The trade-offs did not disappear, they were re-priced: warmup time and hand-kernel engineering instead of build steps and per-SKU artifacts.

Two misconceptions to correct explicitly. "TensorRT-LLM requires building an engine" has been false since 1.0 and is unbuildable on current main; the name now describes ancestry more than mechanism. And "a PyTorch backend means Python-speed serving" is wrong twice over: the hot loop's decisions run in C++ behind bindings, and the GPU-side work runs as captured graphs and fused kernels, so Python remains on the control plane, not the data plane.

Part V continued: Deep dive: in-flight batching and the C++ executor machinery

In-flight batching is NVIDIA's name for continuous batching, the iteration-level scheduling the Orca paper introduced and vLLM popularized: batch membership is reconsidered every model step, so finished sequences leave immediately, waiting requests join immediately, and context (prefill) work for new requests shares iterations with generation (decode) work for old ones.

iteration N:    [A:decode] [B:decode] [C:context chunk 2/3] [D:context chunk 1/4]
B finishes, E arrives
iteration N+1:  [A:decode] [C:context chunk 3/3] [D:context 2/4] [E:context 1/1]
iteration N+2:  [A:decode] [C:decode] [D:context 3/4] [E:decode]

The decision is factored into two cooperating schedulers, and the factoring is worth memorizing because it recurs in every serious engine. The capacity scheduler (batch_manager/capacityScheduler.cpp) answers the memory question: which requests may be in flight at all, given KV cache blocks. Its policy enum (cpp/include/tensorrt_llm/executor/types.h) has three values: kGUARANTEED_NO_EVICT, the default, admits a request only if it can run to completion without eviction; kMAX_UTILIZATION packs more aggressively and accepts that peak memory pressure may force pausing requests (pauseRequests.cpp exists precisely for this); kSTATIC_BATCH reproduces old-style batch-at-a-time behavior for comparison. The micro-batch scheduler (microBatchScheduler.cpp) then answers the compute question: of the admitted requests, what fits this iteration under the token budget, chunking long prompts (ContextChunkingPolicy, first-come-first-served by default) so a 30k-token prefill cannot freeze everyone's decode latency.

Around these two sit the supporting cast, all in cpp/tensorrt_llm/batch_manager/: llmRequest.cpp (request state machine), kvCacheManager.cpp with evictionPolicy.cpp (paged block pool, 32 tokens per block by default, prefix reuse), sequenceSlotManager.cpp (seat assignment), and the cache transceiver files that ship KV blocks between nodes for disaggregated prefill/decode serving. The famous trap: MAX_UTILIZATION is not a free throughput switch; under memory pressure it pauses running requests, which shows up as mid-stream latency spikes, so the default policy's conservatism is a latency guarantee, not timidity. A second trap: the token budget means scheduling is not FIFO fairness; a stream of short requests can interleave ahead of a long prompt's later chunks, which is by design, and tail-latency SLOs are tuned with the scheduler and chunking knobs in the YAML, not by code changes. For the requirements-first version of these ideas, my LLM serving design write-up derives the same trio, continuous batching, paged KV, chunked prefill, from first principles.

Part V continued: Deep dive: quantization co-designed with the hardware

Every engine quantizes; what makes TensorRT-LLM distinctive is that here the format, the kernels, and the silicon are designed by the same company, usually in that order reversed. FP8 arrived with Hopper: the H100's tensor cores execute 8-bit floating point (E4M3 for weights and activations, E5M2 where range matters) natively, so FP8 halves weight memory and roughly doubles matmul throughput relative to FP16 while behaving, from the model's perspective, like a slightly noisier float. Because floats carry exponents, FP8 needs only light per-tensor or per-block scaling rather than the delicate calibration that INT8 required, which is why FP8 became the default serving precision on Hopper almost immediately.

Blackwell extends the bet to four bits with NVFP4. The format: weights are stored as E2M1 (one sign, two exponent, one mantissa bit, representable magnitudes up to 6), in blocks of 16 values that share an FP8 (E4M3) scale, with one FP32 scale per tensor on top. A block is 16 times 4 bits plus 8 bits of scale, 72 bits per 16 values, so about 4.5 bits per weight effective, the same overhead ratio as llama.cpp's Q4 blocks but with a floating-point scale hierarchy the tensor cores consume directly. Compared with the OCP MXFP4 standard (32-value blocks, power-of-two E8M0 scales), NVFP4's smaller blocks and higher-precision scales measurably reduce quantization error at the cost of double the scale overhead, a trade NVIDIA could make unilaterally because it also builds the datapath. On this hardware, low-precision floating point is not an accuracy compromise bolted on afterwards; it is the operating point the chips were designed around, and this repository is where that intent becomes runnable kernels, in the FP4 GEMMs and fused attention under cpp/tensorrt_llm/kernels/ and cutlass_extensions/.

Operationally, quantized checkpoints usually come from the TensorRT Model Optimizer toolkit, and NVIDIA publishes pre-quantized FP8 and FP4 versions of popular open models on Hugging Face, so serving one is a model-name change rather than a calibration project; the loading side lives in tensorrt_llm/quantization/ and the per-model quant configs in the PyTorch backend. The KV cache can be quantized to FP8 as well, which matters because at high concurrency the cache, not the weights, dominates memory. The trap to correct: 4-bit weights do not mean the whole computation is 4-bit, activations flow at higher precision with scaling at the boundaries; and the gains are architecture-gated, so an FP4 checkpoint does nothing for you on Ampere. Check what your GPU generation actually accelerates before choosing a checkpoint.

Part VI: Reading the repository

Two codebases share the roof: the Python package tensorrt_llm/ and the C++ core cpp/tensorrt_llm/. All paths verified on main, July 2026; this repo moves fast, so trust the layering over the exact file names.

Stage 0: orientation (one evening)

Read the README and the quick start in docs/, then run the LLM API example in the release container. Questions you should answer: what are the two backends and which is default? What three commands does the wheel install? Why does the first generation take so long?

Stage 1: the public surface

Read tensorrt_llm/llmapi/llm.py, skim llmapi/llm_args.py (do not read it linearly; search it when you meet a knob), then commands/serve.py and serve/openai_server.py with openai_protocol.py. Questions: where does the chat template get applied? How does a YAML options file become LLM arguments? What does the server hand downward, text or tokens?

Stage 2: the executor boundary

Read tensorrt_llm/executor/: executor.py, proxy.py, worker.py, ipc.py, request.py, result.py, postproc_worker.py. Questions: which process owns the GPU? What crosses the IPC boundary and in which direction? Why do detokenization and streaming live in separate worker processes?

Stage 3: the PyExecutor

The heart. In tensorrt_llm/_torch/pyexecutor/ read py_executor.py (find the loop variants), executor_request_queue.py, llm_request.py, scheduler/scheduler.py, resource_manager.py, model_engine.py, cuda_graph_runner.py, and sampler/. Questions: what happens in one iteration, in order? What does the overlap loop overlap? Which decisions are delegated to C++ and through what?

Stage 4: the C++ core

In cpp/tensorrt_llm/batch_manager/ read capacityScheduler.cpp, microBatchScheduler.cpp, kvCacheManager.cpp, llmRequest.cpp, and evictionPolicy.cpp, with the executor API types in cpp/include/tensorrt_llm/executor/ beside them, and glance at cpp/tensorrt_llm/nanobind/ to see how Python reaches all of it. Questions: how do the three capacity policies differ in code? How is a KV block found for reuse? What exactly does ScheduledRequests contain when it returns to Python?

Stage 5: models and kernels

Read one architecture in _torch/models/ and the modules it uses in _torch/modules/ and _torch/attention_backend/, then descend into cpp/tensorrt_llm/thop/ to see kernel registration and sample a kernel family in cpp/tensorrt_llm/kernels/. Questions: how does a plain-looking PyTorch module end up calling a CUTLASS FP8 GEMM? Where do quantization scales live at runtime?

Where not to start: cutlass_extensions/ and the vendored kernel trees (deep_gemm, flash_mla, deep_ep), which assume fluency in CUTLASS; 3rdparty/; triton_backend/; the _torch/auto_deploy/ experimental compiler; and the C++ runtime/ engine-loading tree, which is best read last, as history, the way Part IV frames it.

Part VII: Hands-on labs

All labs assume an NVIDIA GPU and the release container. Numbers vary with GPU, model, and version; treat them as shapes.

Lab 1: watch warmup earn its keep. Run the LLM API quickstart with logging raised:

TLLM_LOG_LEVEL=INFO python quickstart.py

Observe the startup sequence in the logs: weight loading, KV cache pool sizing, autotuning, CUDA graph capture per batch size, and then how much faster the second llm.generate() call is than the first. This is the Part V argument, compile-time work re-priced as warmup, visible in timestamps.

Lab 2: in-flight batching under load. Serve a small model, then hit it with many concurrent streams:

trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" &
for i in $(seq 1 16); do
  curl -s http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0",
         "messages":[{"role":"user","content":"Write 100 words about oceans."}]}' \
    > /dev/null &
done; wait

Watch nvidia-smi utilization and the iteration statistics in the server log. Then repeat the 16 requests serially and compare wall-clock totals: the concurrent run finishing in a small multiple of the serial single-request time, rather than 16 times it, is continuous batching measured.

Lab 3: scheduler policy as an observable. Create two YAML files differing only in scheduler_config capacity policy (GUARANTEED_NO_EVICT vs MAX_UTILIZATION), constrain KV memory with a low free_gpu_memory_fraction, and drive both with trtllm-bench at a request rate high enough to saturate. Measure throughput and per-request latency percentiles. Expect the aggressive policy to gain throughput and pay for it in tail latency when pausing kicks in, the exact trade-off from the deep dive.

Lab 4: prefix reuse. With enable_block_reuse: true, send a request with a 2,000-token system prompt, then a second request sharing that system prompt with a different question, and compare time-to-first-token in the responses' timing (or the perf metrics endpoint if enabled). The second request's prefill should shrink dramatically: KV blocks from stage 7 being found instead of computed.

Lab 5: what FP8 buys (Hopper or newer required). Serve the same model twice, once from the standard BF16 checkpoint and once from NVIDIA's pre-quantized FP8 variant on Hugging Face, and run identical trtllm-bench sweeps. Record weight memory (nvidia-smi after load), max sustainable request rate, and an accuracy spot-check with trtllm-eval. You are measuring the co-design claim rather than taking it on faith; on pre-Hopper hardware this lab intentionally fails, which is itself the lesson about architecture-gated formats.

Lab 6: find the boundary. No GPU needed. In the repo, start from pyexecutor/scheduler/scheduler.py's BindCapacityScheduler, find the binding it constructs, locate the C++ implementation in batch_manager/capacityScheduler.cpp, and read one policy's admission loop end to end. Write down, in one paragraph, what crosses the language boundary per iteration. This is the single most clarifying exercise in the codebase.

Part VIII: Questions and model answers

1. What is TensorRT-LLM in one sentence? NVIDIA's open source serving engine for its own GPUs: a PyTorch front half defining models and orchestration, a C++ back half making per-iteration scheduling and memory decisions, and kernels that exploit NVIDIA-specific numeric formats like FP8 and NVFP4.

2. What changed at the 1.0 release? The PyTorch-native backend became the stable default and the LLM API became the stable surface, displacing the classic convert-then-trtllm-build engine workflow, which was subsequently removed from the shipped tooling. Models now load directly from Hugging Face checkpoints with no offline compile step.

3. What exactly was a TensorRT engine? A serialized execution plan produced ahead of time: operator fusion decided, one measured-best kernel tactic recorded per op, precisions fixed, memory statically planned, all specific to one GPU architecture and library version. The runtime replayed it rather than interpreting a graph.

4. If the compiler flow was removed, where did its ideas go? Into three surviving mechanisms: CUDA graph capture, which records replayable decode iterations; the warmup autotuner, which is tactic selection deferred to startup; and hand-fused kernels, which are the fusion pass applied where it pays. The trade moved from build-time rigidity to warmup cost.

5. Trace a chat request through the layers. FastAPI protocol layer applies the chat template and calls the LLM API; the generation executor proxies the request over IPC to a worker process owning the GPU; the PyExecutor loop schedules it via the C++ capacity and micro-batch schedulers, allocates paged KV blocks, runs the PyTorch model engine under CUDA graphs, samples, and streams tokens back through post-processing workers to the SSE response.

6. Why is the serving loop split across processes? To keep the GPU-feeding process isolated from Python-side stalls: the API frontend, detokenization, and result post-processing run in their own processes, connected by ZeroMQ-based IPC, so GIL contention and slow clients cannot starve the iteration loop. Multi-GPU ranks additionally run under MPI.

7. What are the two schedulers and why two? The capacity scheduler decides which requests may be in flight given KV memory (a per-request admission question), and the micro-batch scheduler decides what work from those requests runs this iteration under the token budget (a per-step packing question). Separating memory admission from compute packing lets each policy vary independently.

8. Compare GUARANTEED_NO_EVICT and MAX_UTILIZATION. The default admits a request only when its worst-case KV usage can be satisfied to completion, so latency is predictable and nothing is paused. MAX_UTILIZATION packs more requests and accepts that memory pressure may pause running requests mid-stream, buying throughput with tail latency. STATIC_BATCH exists mainly as a baseline.

9. How does the KV cache work here? It is paged: fixed-size blocks of 32 tokens by default, drawn from a pool sized as a fraction of post-load free GPU memory, with prefix-matched block reuse across requests and an eviction policy over the pool. The bookkeeping is C++ (kvCacheManager.cpp), reached from Python through bindings.

10. What makes FP8 easier to deploy than INT8 was? Floats carry per-value exponents, so FP8 tolerates the dynamic range of transformer activations with only light scaling, where INT8 needed careful calibration and often per-channel tricks to avoid outlier damage. With Hopper executing FP8 natively in tensor cores, it halves weight memory and raises matmul throughput at near-baseline accuracy.

11. Describe NVFP4 precisely. Elements are E2M1 four-bit floats; every 16 elements share an FP8 E4M3 scale, and each tensor carries an FP32 scale on top, totaling about 4.5 bits per weight. Against MXFP4's 32-element blocks with power-of-two scales, the finer blocks and richer scales cut quantization error, and Blackwell tensor cores consume the format natively.

12. When is vLLM the better choice? When portability across hardware, ecosystem breadth, or operational simplicity outweigh the last measure of NVIDIA-side performance, which is often. TensorRT-LLM earns its complexity when you are committed to NVIDIA fleets, want FP8/FP4 at day one on new silicon, or deploy behind Triton with NVIDIA's supported path.

13. A server OOMs at startup though the model fits in VRAM. What is happening? The KV cache manager pre-allocates a large fraction of post-load free memory by design, and other consumers (CUDA graphs, fragmentation, another process) can push the total over. Lower kv_cache_config.free_gpu_memory_fraction or reduce max batch/sequence limits; do not conclude the model is too large.

14. Throughput is fine but p99 latency spikes mid-generation. Where do you look first? Scheduler policy and memory pressure: under MAX_UTILIZATION, paused-and-resumed requests manifest exactly this way. Check iteration stats for paused requests, then either move to the no-evict policy or give the KV pool more headroom, and check whether giant prompts are being chunked sensibly.

15. Why does the first request after deploy take so long, and what do you do about it? Warmup: kernel autotuning and CUDA graph capture happen lazily at startup and on first shapes. It is expected, and production deployments absorb it by warming the server with representative traffic before adding it to the load balancer, not by disabling the optimizations.

16. What runs in Python and what runs in C++ in the default backend? Python: the API surface, the executor loop's orchestration, model definition, and configuration. C++: capacity and micro-batch scheduling algorithms, KV block management, and all kernels, exposed through nanobind bindings and torch custom ops. Python steers; C++ and the GPU do everything counted per token.

Part IX: Design lessons

Compile-ahead loses to libraries when the workload outruns the compiler. Whole-model AOT compilation was the right call when architectures changed yearly and the wrong one when they changed monthly; the durable wins (fusion, tuned kernel choice, replayable plans) survived by being repackaged at smaller granularity. The same arc played out in databases (whole-query codegen vs vectorized interpreters) and graphics (monolithic shaders vs pipeline caches).

Keep Python on the control plane, never the data plane. The engine's answer to "Python is slow" is not rewriting in C++ wholesale but drawing the boundary carefully: per-iteration decisions and kernels below the bindings, orchestration above. Every high-performance Python system that works, from NumPy to PyTorch itself, is this one pattern applied consistently.

Process isolation is a scheduling tool. Proxy, worker, and post-processing processes exist so the GPU never waits on the GIL, a slow client, or detokenization. Nginx workers, browser process models, and database background writers all encode the same insight: isolate the latency-critical loop from everything with unpredictable timing.

Co-design up and down the stack when you own it. NVFP4 exists because the format designers and the tensor-core designers were the same company; the software ships kernels for silicon on day one, and the silicon implements formats the software proved out. Apple's silicon-plus-frameworks story and Google's TPU-plus-XLA story are the same play; the cost, as always, is that the advantage does not travel.

Make policy an enum, not a fork. The capacity scheduler's three policies encode a genuine latency-throughput trade as a configuration value with a conservative default, letting operators choose without patching code. Databases do this with isolation levels, kernels with I/O schedulers; naming the trade-off is half of managing it.

Part X: Memorization framework

One sentence to keep: TensorRT-LLM serves OpenAI-style requests through a Python-orchestrated, C++-scheduled executor loop that batches at iteration granularity into a PyTorch model running CUDA-graphed, hand-fused kernels on NVIDIA's own low-precision formats.

Request → Protocol → LLM API → IPC → PyExecutor → Scheduler(C++) → KV blocks
        → Model engine → Kernels → Sampler → Stream

The chain mapped to real paths:

Protocol    tensorrt_llm/serve/openai_server.py, openai_protocol.py
LLM API     tensorrt_llm/llmapi/llm.py, llm_args.py
IPC         tensorrt_llm/executor/ (proxy.py, worker.py, ipc.py)
PyExecutor  tensorrt_llm/_torch/pyexecutor/py_executor.py
Scheduler   pyexecutor/scheduler/ → cpp/tensorrt_llm/batch_manager/
            capacityScheduler.cpp, microBatchScheduler.cpp
KV blocks   batch_manager/kvCacheManager.cpp (32 tokens/block default)
Engine      _torch/pyexecutor/model_engine.py, _torch/models/, cuda_graph_runner.py
Kernels     cpp/tensorrt_llm/kernels/, cutlass_extensions/, thop/
Sampler     _torch/pyexecutor/sampler/
Stream      executor/result.py, postproc_worker.py → SSE

Memorize these:

• Since 1.0: PyTorch backend default, LLM API stable, no engine build step; trtllm-serve, trtllm-bench, trtllm-eval are the three shipped commands.
• Two schedulers: capacity (memory admission; NO_EVICT default) then micro-batch (per-iteration packing with chunked context).
• In-flight batching = continuous batching: membership reconsidered every iteration; prefill and decode share iterations.
• FP8 = Hopper, E4M3, light scaling. NVFP4 = Blackwell, E2M1 in 16-value blocks with E4M3 scales plus FP32 per tensor, ~4.5 bits effective.
• The compiler survives as: CUDA graphs (plan replay), autotuner (tactic selection at warmup), hand-fused kernels (the fusion pass).

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site covers the same ground in depth, the companion link points there.

1. Yu et al., Orca, A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. The paper that introduced iteration-level scheduling, the idea this repository ships as in-flight batching in the C++ batch manager.
2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The paged KV cache design behind the block pools in kvCacheManager.cpp, covered in the vLLM walkthrough.
3. Agrawal et al., SARATHI, Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills, 2023. The chunked-prefill idea the micro-batch scheduler applies so long prompts share iterations with decode work. The LLM serving design write-up on this site derives it from requirements.
4. Zhong et al., DistServe, Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, OSDI 2024. The case for splitting prefill and decode onto separate machines, which is what the cache transceiver files in the batch manager exist to serve.
5. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The IO-aware fused-attention approach the TRT-LLM attention kernels descend from, covered in the flash-attention walkthrough.
6. Micikevicius et al., FP8 Formats for Deep Learning, 2022. The NVIDIA, Arm, and Intel spec for E4M3 and E5M2, the formats Hopper executes natively. The low-precision groundwork lives in the mixed precision note on this site.
7. Rouhani et al., Microscaling Data Formats for Deep Learning, 2023. The OCP block-scaled MX formats, including the MXFP4 that Part V contrasts NVFP4 against.
8. NVIDIA, Pretraining Large Language Models with NVFP4, 2025. The fullest public description of the NVFP4 format and the recipes that make four-bit floats workable.
9. Xiao et al., SmoothQuant, Accurate and Efficient Post-Training Quantization for Large Language Models, 2022. The INT8 activation-outlier calibration story whose difficulty explains why FP8 won so quickly.
10. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022. The draft-and-verify sampling idea behind the speculative decoding knobs, and one of the dynamic behaviors that strained the static-graph era.
11. Shoeybi et al., Megatron-LM, Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. The tensor parallelism behind tp_size, covered in the Megatron-LM walkthrough.

Part XII: Final takeaway