Design an LLM serving platform

This problem usually gets drawn as a web app, a chat service in front of a box labeled LLM, and the standard ChatGPT reference design does exactly that, a stateless chat service, Postgres for chats and messages, and an inference service reached through a queue. That skeleton is correct and this article builds it. The design only becomes interesting inside the model box, though, because a transformer generating text is a strange workload. It produces one token at a time, and every one of those steps has to re-read every weight in the model, which makes generation bound by memory bandwidth rather than arithmetic and turns the whole platform into an exercise in scheduling around a cache.

Two numbers govern everything. Time to first token (TTFT) is how long the user stares at nothing before the response starts, and it is spent in queueing plus one compute-heavy pass over the prompt. Inter-token latency (ITL) is the pace of the stream after that, and it is spent in a memory-bound loop the server shares across every active request. Nearly every technique in this article, continuous batching, PagedAttention, speculative decoding, quantization, exists to move one of those two numbers or to lower the GPU bill while holding them.

The walkthrough sizes the KV cache for a real model, Llama 3.1 8B, works through the two big scheduling ideas from the Orca and vLLM papers, and then assembles the ordinary but load-bearing platform around the engine, streaming over SSE, chat history in Postgres, a per-run token log in Redis, routing across model pools, and autoscaling on GPUs that take minutes to warm.

The one-line version. The one-line version. An LLM serving platform is a memory-bandwidth scheduler wearing a web app's clothes. Prefill is compute-bound and sets TTFT, decode re-reads all 16 GB of weights per token and sets inter-token latency, so throughput comes from amortizing each weight read across a batch of sequences. Continuous batching keeps the batch full at iteration granularity, PagedAttention keeps the KV cache dense enough to admit the batch, and speculative decoding, quantization, routing, and autoscaling are all negotiated against the two SLOs.

Scope and requirements

Functionally the product is a chat surface. A signed-in user sends a message in a conversation, the reply streams in token by token, history persists across devices, and the same API fronts more than one model, a large flagship and a small cheap one at minimum. In scope are the serving path from prompt to streamed reply, the storage for conversations, and the GPU fleet that runs the models. Out of scope are training and fine-tuning, retrieval augmentation, and the safety stack, each a system of its own.

Non-functionally, two latency numbers are the contract. Time to first token is the delay between send and the first visible character, and users read it as whether the product works at all, so it gets a tight budget, a few hundred milliseconds at p95 for the interactive tier. Inter-token latency is the gap between tokens after that, and it only has to beat reading speed to feel instant, which puts it in the tens of milliseconds. The pair matters because they are bought with different resources. TTFT is bought with compute and queue discipline, ITL is bought with memory bandwidth, and both trade against throughput, since packing more requests onto a GPU makes every individual stream a little slower.

Streaming is not a nicety. A 500-token answer at 30 milliseconds per token takes 15 seconds to finish, and nobody waits 15 seconds on a spinner, so the reply must render as it is generated, which reaches back through the whole stack. The transport must push increments, the chat service must not buffer, and the failure story must cope with a stream dying halfway through an answer. Cost closes the requirements. The dominant line item is GPU hours, so the design metric under everything is tokens per second per GPU at a fixed SLO, and a technique that doubles it halves the bill.

The two SLOs are time to first token and inter-token latency, they are bought with different resources, and every throughput trick in the design is negotiated against them.

Prefill, decode, and the bandwidth wall

A transformer answers in two phases with opposite characters. Prefill takes the entire prompt and runs it through the model in one pass, and because every prompt token can be processed in parallel, the work arrives as large matrix-matrix multiplies that saturate the tensor cores. Prefill is compute-bound, it scales with prompt length, it writes the key and value vectors for every prompt token into the KV cache, and it ends by emitting the first output token, which is why TTFT is essentially queue time plus prefill time.

Decode is the opposite. Generation is autoregressive, token N+1 depends on token N, so each sequence advances one token per forward pass, and a forward pass for one new token is a chain of matrix-vector products. The arithmetic is tiny, roughly two floating point operations per parameter, about 16 GFLOPs for an 8B model, which the tensor cores finish in microseconds. The traffic is not tiny. Every step must stream every weight from HBM into the compute units, all 16 GB of a bf16 Llama 3.1 8B, and an H100 SXM moves 3.35 TB/s, so the loop bottoms out near 3.35 TB/s divided by 16 GB, around 200 steps per second, about 5 milliseconds per token, no matter how idle the roughly 990 dense teraFLOPS of FP16 tensor compute sits, a figure the datasheet quotes as 1,979 with sparsity. A single user decoding on an 80 GB H100 uses a fraction of a percent of its arithmetic and all of its memory bus.

The escape is that the weight read is shareable. If 60 sequences sit in the batch, one 16 GB sweep advances all 60, and the per-token cost falls almost 60-fold before KV cache reads and compute start to matter. That single fact shapes the entire platform. Throughput is bought with batch size, and batch size is bought with KV cache memory, so the engineering below is mostly about keeping the batch full, keeping the KV cache dense, and stopping the two phases from trampling each other, because a multi-second prefill dropped into the middle of the decode loop stalls the inter-token latency of every stream on the GPU. Engines mitigate that by slicing prefill into chunks interleaved with decode steps, and TensorRT-LLM goes further with prefill-decode disaggregation, running the two phases on separate worker pools so a burst of long prompts cannot freeze the live streams.

Decode at batch size one wastes the GPU, so the platform's real job is to amortize each 16 GB weight sweep across as many concurrent sequences as the KV cache will admit.

Sizing the KV cache for Llama 3.1 8B

The KV cache exists because attention looks backward. Each new token attends over the keys and values of every token before it, and recomputing those from scratch at every step would turn each decode step into a fresh prefill, so the engine stores them once and reuses them. The cost is memory that grows linearly with every token of every active sequence, and that memory is the currency the batch is bought with.

The config.json for Llama 3.1 8B Instruct supplies the numbers. 32 hidden layers, 32 attention heads but only 8 key-value heads thanks to grouped-query attention, hidden size 4096 so each head is 128-dimensional, and a 131,072-token context window. A token's cache entry is one key vector and one value vector per KV head per layer, so 2 × 32 × 8 × 128 values, 65,536 of them, at 2 bytes each in bf16. 128 KB per token.

Two consequences drop out. First, a single request at the full 131,072-token window costs 131,072 × 128 KB, which is exactly 16 GiB, the same as the model weights, so one maxed-out conversation doubles the model's footprint. Second, on an 80 GB H100 with 16 GB of weights and a few GB of activations and runtime overhead, roughly 60 GB is left for KV, call it 480,000 tokens, which is about 60 concurrent conversations at 8K tokens each. That is the batch the bandwidth argument was begging for, and it exists only because of grouped-query attention. With full multi-head attention the per-token cost would be four times larger, and the vLLM paper's example model, OPT-13B, which predates the technique, needed 800 KB per token and up to 1.6 GB of cache for a single 2,048-token request.

# KV cache per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
layers, kv_heads, head_dim = 32, 8, 128        # Llama 3.1 8B config.json
bytes_bf16 = 2
per_token = 2 * layers * kv_heads * head_dim * bytes_bf16
print(per_token)                               # 131,072 B = 128 KB per token

context = 131_072                              # max_position_embeddings
print(per_token * context / 2**30)             # 16.0 GiB, same as the weights

hbm, weights, overhead = 80, 16, 4             # H100 SXM budget, GiB
kv_budget = (hbm - weights - overhead) * 2**30
print(kv_budget // per_token)                  # ~491k tokens -> ~60 chats at 8K

128 KB per token means KV memory, not weights, decides how many conversations fit on a GPU, so every KV byte saved converts directly into batch size and therefore throughput.

Inside the engine: continuous batching and PagedAttention

The first generation of serving systems batched at request granularity. A batch formed, ran until every sequence in it finished, and only then admitted new work, which is a disaster for a workload where one reply is 10 tokens and its neighbor is 1,000, because finished sequences hold their slots as dead weight while arriving requests wait on the stragglers. Orca, from OSDI 2022, replaced that with iteration-level scheduling. The scheduler invokes the engine for exactly one forward pass at a time, and at every iteration boundary finished sequences leave the batch and waiting sequences join it, so the batch is continuously refilled instead of drained. With selective batching to let sequences of different lengths share one kernel launch, Orca measured 36.9× the throughput of NVIDIA's FasterTransformer at the same level of latency. Every serious engine adopted the idea, vLLM under the name continuous batching, TensorRT-LLM as in-flight batching.

Continuous batching then hits a memory problem. If sequences join and leave constantly, how do you lay out a cache that grows token by token and dies at unpredictable times? Pre-vLLM systems allocated each request one contiguous slab sized for the maximum possible sequence length, because the kernels wanted contiguous tensors, and the vLLM paper measured the result, only 20.4 to 38.2 percent of KV cache memory actually held token states, the rest lost to internal fragmentation inside oversized slabs, external fragmentation between them, and reservations for tokens never generated. PagedAttention imported virtual memory's answer. The cache is cut into fixed blocks of 16 tokens, a per-sequence block table maps logical positions to physical blocks scattered anywhere in HBM, and the attention kernel follows the table. Waste collapses to at most one partially filled block per sequence, near zero, and the reclaimed memory becomes batch size, which is where the paper's 2 to 4× throughput gain over FasterTransformer and Orca comes from. Nothing about the math got faster. More sequences fit.

Block tables also make sharing trivial, because two sequences whose prompts start identically can point their tables at the same physical blocks. That matters commercially, since real traffic is saturated with shared prefixes, the system prompt every request carries, the conversation history that turn N+1 re-sends after turn N, few-shot templates. vLLM exposes this as prefix caching, and SGLang builds its engine around it with RadixAttention, which keeps all live prefixes in a radix tree so any new request automatically reuses the longest cached prefix, one reason SGLang reports serving trillions of tokens a day across more than 400,000 GPUs. For a chat platform this is the difference between re-prefilling 4,000 tokens of history every turn and prefilling only the user's newest 50-token message, which is most of the TTFT budget on a long conversation.

One replica's serving internals. The batcher reschedules at every forward pass, prefill writes KV blocks and produces the first token, the decode loop advances every running sequence one token per iteration against the paged KV cache, and an optional draft model lets the target verify several tokens per weight sweep.

Orca keeps the batch full by rescheduling at every forward pass, PagedAttention keeps the KV cache dense with 16-token blocks and block tables, and prefix reuse turns re-sent conversation history from a TTFT tax into a cache hit.

The platform around the engine

Around the engine sits an ordinary web system, and the standard ChatGPT reference design draws it cleanly. A web client talks through an API gateway to a stateless chat service, chats and messages live in Postgres, and the model runs in a separate inference service. The statelessness is deliberate. The chat service holds no conversation in memory, it loads history from Postgres, assembles the prompt, and any replica can serve any user, which lets the web tier scale independently of the GPU tier. The first version wires the chat service to the inference service with a synchronous HTTP call, and it works until the first burst, because a request that arrives while every GPU is busy has nowhere to wait except an open connection, and a generation that takes 30 seconds pins a thread for 30 seconds.

The final design decouples the two with a queue and a log. The chat service persists the user message, creates a generation run, and enqueues it on a priority queue. Workers pull runs and batch them into the inference service, whose model workers run with prefix caching so re-sent history is cheap. As tokens come out, the worker appends each one to a Redis Stream keyed run:{runId}, and the chat service tails that stream and forwards tokens to the client over SSE. Postgres keeps three tables, chats, messages, and generation runs mapping runId to chatId and messageId, so a run is a first-class record rather than an in-flight HTTP call.

The Redis Stream is the piece that earns its keep on failure. Because every token lands in a log with a sequence number, the client's SSE connection and the generation itself have independent lifetimes. A phone that drops off wifi mid-answer reconnects, presents the last event id it saw, and the chat service replays the stream from that sequence and then continues live, no tokens lost and no regeneration paid for. The same log lets a second device attach to a run in progress, and it lets the chat service persist the final assistant message by reading its own log rather than trusting a worker to report back. SSE fits the transport exactly, the flow is one-way, browsers reconnect automatically, and Last-Event-ID gives resume for free, the same reasoning that carried the price stream in the brokerage design.

# Chat service: serve (or resume) the token stream for one generation run.
# The client's Last-Event-ID header becomes last_seq, so a dropped SSE
# connection replays missed tokens from the Redis Stream, then goes live.
def stream_run(run_id, last_seq="0"):
    while True:
        entries = redis.xread({f"run:{run_id}": last_seq}, block=15_000)
        for seq, fields in entries:
            last_seq = seq
            if fields["type"] == "done":
                return                     # run finished, close the stream
            yield f"id: {seq}\nevent: token\ndata: {fields['text']}\n\n"

The standard ChatGPT service shape. The chat service persists the message, enqueues a run, workers batch prompts into the inference service, every generated token is appended to a per-run Redis Stream, and the chat service tails the stream out to the client over SSE.

Statelessness in the chat tier plus a per-run token log in Redis means the generation, the SSE connection, and the durable record in Postgres can each fail independently and reconverge.

Making tokens cheaper: speculative decoding and quantization

Speculative decoding attacks the serial nature of decode directly. A small draft model, cheap because reading its weights is cheap, proposes the next k tokens, and the big target model scores all k in a single forward pass, which costs roughly one token's worth of weight traffic because the weights are read once regardless of how many positions are verified. Accepted tokens are kept, the first rejection is resampled from the target's own distribution, and the rejection-sampling construction in Leviathan, Kalman, and Matias's paper guarantees the output distribution is exactly the target model's, not an approximation. They measured 2× to 3× on T5-XXL. The catch is that the win depends on the draft agreeing with the target, easy prose accepts most tokens while code and math accept fewer, and on the server having idle compute, because on a fully batch-saturated GPU the flops spent drafting and verifying were flops that batching would have spent better. Speculation is a latency tool for interactive tiers, not a free throughput multiplier.

Quantization attacks the 16 GB itself. The ladder runs from bf16 to FP8 weights and activations, which vLLM ships as FP8 W8A8 through LLM Compressor, then to 4-bit weight-only formats like AWQ and GPTQ, INT4 W4A16 in vLLM's taxonomy, each rung shrinking the bytes a decode step must stream, which at small batch converts almost directly into inter-token latency. Weight-only 4-bit is aimed precisely at the bandwidth wall, weights shrink while activations stay high precision, and its advantage fades at very large batch where the workload tips back toward compute. The KV cache can be quantized separately, vLLM lists quantized KV cache as its own feature, and after the sizing section it is obvious why, FP8 KV doubles the token capacity of the same 60 GB and therefore the admissible batch. The discipline that must travel with all of it is evaluation, because a quantized model is a different model, and it ships only after task-level quality checks, not after eyeballing perplexity.

Speculation spends spare compute to compress serial decode steps without changing outputs, quantization shrinks the bytes each step must stream, and both are levers on the same memory-bandwidth bill.

Routing, multi-model serving, and autoscaling on GPU

In front of the engine replicas sits a router with three jobs. It maps model names to replica pools, since the flagship and the cheap model run on different hardware footprints and scale independently. It keeps session affinity, pinning a conversation to the replica that already holds its prefix in cache, because a radix-tree hit on 4,000 tokens of history is worth more than perfect load spreading, with spillover when the pinned replica runs hot. And it enforces admission control with priorities, interactive requests ahead of batch jobs, per-tenant token budgets, and load shedding that rejects early at the router rather than letting requests time out deep in a queue. The priority queue from the platform section is where the tiers meet, and a batch tier that tolerates minutes of delay is what soaks up the fleet's spare capacity overnight.

Autoscaling on GPUs breaks the reflexes learned on stateless web tiers. Utilization percentages are the wrong signal, because a replica can look busy while its true constraint is KV memory, and the signals that actually track saturation are the ones the engine exposes, KV cache utilization, queue depth, and the SLOs themselves, with a rising TTFT p95 as the earliest honest sign that admission has outrun capacity. Scale-out is slow. A new replica must be scheduled onto a scarce instance type, pull 16 GB of weights, initialize the engine, and warm its caches, which takes minutes rather than the seconds a stateless service takes, so the fleet scales ahead of the daily traffic curve instead of reacting to it, keeps a warm buffer for spikes, and treats scale-to-zero as a cold start measured in minutes. Draining is equally physical. A replica with 60 live streams cannot be killed, it must stop admitting and let its sequences finish.

Route for cache affinity, admit by priority, and scale on KV utilization, queue depth, and TTFT, on a fleet where adding a replica takes minutes and removing one requires a drain.

Alternatives that also work, and non-starters

The engine choice is genuinely open. vLLM is the default open-source pick and the source of PagedAttention. SGLang is a peer, strongest where prefix reuse dominates, agent loops and heavy shared system prompts, and it runs at enormous production scale. TensorRT-LLM trades portability for peak NVIDIA performance, with compiled custom kernels for attention, GEMMs, and MoE, in-flight batching, speculative decoding, and prefill-decode disaggregation, at the cost of a build step and NVIDIA lock-in. All three implement the same core ideas, continuous batching, paged KV, prefix caching, quantization, speculation, which is itself the lesson, the ideas are the load-bearing content and the engines are interchangeable carriers. The other honest alternative is not building the platform at all, because below sustained utilization the per-token price of a hosted API beats owning idle GPUs, and the build decision should follow the utilization math rather than pride.

The non-starters are measured, not stylistic. Request-level static batching is out, and Orca's 36.9× is the measurement of what it leaves on the table. Contiguous max-length KV allocation is out, since 20.4 to 38.2 percent memory utilization means surrendering more than half the batch. A blocking, non-streaming API is out for interactive use, because a 15-second silent wait reads as an outage and pins connections besides. Autoscaling on utilization counters is out because the binding resource is KV memory and bandwidth, not the number the dashboard makes convenient. And holding conversation state in chat-service memory is out, it welds users to replicas, breaks on every deploy, and buys nothing that Postgres plus prefix caching does not provide better.

Pick any of vLLM, SGLang, or TensorRT-LLM and keep the ideas, but static batching, contiguous KV slabs, blocking APIs, and utilization-based autoscaling are quantified mistakes.

Questions and answers

The core ideas as questions with the answers given outright. Each wrong multiple-choice option is marked with why it is wrong, and the ordering ones show the correct sequence.

1A single H100 SXM serves Llama 3.1 8B in bf16 to one user, decode runs at about 200 tokens per second, and the tensor cores are nearly idle. What is the binding constraint?

✓Every decode step must stream all 16 GB of weights from HBM, and 3.35 TB/s of bandwidth divided by 16 GB caps the loop near 200 steps per second
✗The GPU lacks the floating point throughput to run the model any faster at batch size one. An 8B model needs about 16 GFLOPs per token against roughly 990 dense teraFLOPS of FP16 tensor compute, so arithmetic is nowhere near the limit at batch one.
✗Python and framework overhead in the serving loop dominates the step time. Overhead is real but engines with fused CUDA kernels hit the same roughly 5 ms floor, because the floor is weight traffic, not host code.
✗Reading the KV cache dominates the step time even for short conversations. At short context the KV read is tiny next to 16 GB of weights, and KV traffic only rivals weight traffic near the full 128K window.

Why: The reflex is to blame arithmetic, but decode's math is trivial, about two FLOPs per parameter per token, roughly 16 GFLOPs, which the tensor cores finish in microseconds. The step cost is dominated by streaming the entire weight set from HBM on every forward pass, and 3.35 TB/s over 16 GB gives roughly 200 sweeps per second, about 5 ms per token. That is exactly why batching is the throughput lever, one sweep advances every sequence in the batch.

2On Llama 3.1 8B in bf16, a single conversation at the full 131,072-token context window consumes about as much GPU memory for its KV cache as the model weights themselves.

✓True
✗False. 128 KB per token times 131,072 tokens is exactly 16 GiB, and 8B parameters at 2 bytes each is about 16 GB of weights, so the two really are equal.

Why: It sounds absurd that bookkeeping could match the model, but the per-token entry is 2 × 32 layers × 8 KV heads × 128 dims × 2 bytes, 128 KB, and at the full window that multiplies out to 16 GiB against about 16 GB of weights. It would be four times worse without grouped-query attention, which is why KV memory, not weights, is the resource the batch is bought with.

3Put the life of one request through a continuous-batching engine in order.

The request lands on a replica and waits until the scheduler sees enough free KV blocks to admit it
Prefill runs the whole prompt in one compute-bound pass and writes its KV blocks
The first token streams out, stopping the TTFT clock
The sequence joins the running batch and decode advances it one token per engine iteration
The sequence finishes, leaves the batch at the next iteration boundary, and its freed KV blocks admit waiting requests

Why: Admission happens against KV block availability, not CPU or GPU counters, because blocks are the resource the request will consume. Prefill runs once and sets TTFT, then the sequence advances one token per engine iteration alongside every other running sequence, and it exits at an iteration boundary so its blocks immediately admit waiting work. Request-level batching would instead hold the slot until the whole batch drained, which is the waste Orca measured.

4Before vLLM, serving systems pre-allocated each request's KV cache as one contiguous chunk sized to the maximum possible sequence length. What did the vLLM paper measure about that scheme, and what did PagedAttention change?

✗It made the attention kernel itself 2 to 4 times faster by reordering the computation. The kernel is not faster, the win is memory, and the larger feasible batch is what delivers the 2 to 4× throughput.
✓Only 20.4 to 38.2 percent of KV cache memory held actual token states, and PagedAttention's 16-token blocks with per-sequence block tables cut waste to near zero, letting far more sequences fit per GPU
✗It compressed the KV cache to lower precision, halving its size at some quality cost. Lower precision is KV cache quantization, a separate and orthogonal feature. PagedAttention changes layout, not precision.
✗It moved cold KV blocks out to CPU RAM, paging them back in on demand. The paging vocabulary is borrowed from operating systems, but the blocks stay in HBM. It is indirection, not swapping to host memory.

Why: The measurement was about memory, not math. Contiguous max-length slabs meant only 20.4 to 38.2 percent of KV memory held real token states, the rest lost to internal fragmentation, external fragmentation, and reservations for tokens never generated. PagedAttention's block tables reclaim that waste so more sequences fit per GPU, which is what delivers the 2 to 4× throughput at the same latency. The kernel does the same attention arithmetic through one level of indirection.

5Speculative decoding runs a small draft model ahead of the large target model. What is the actual contract, and when does it pay?

✗It trades a small amount of output quality for speed, like quantization does. The rejection-sampling verification keeps the target distribution exactly, so quality is unchanged by construction, unlike quantization.
✗It speeds up prefill and decode about equally, since both run through the same weights. Prefill is already parallel and compute-bound. Speculation targets the serial decode loop only.
✓The draft proposes k tokens, the target verifies all k in one forward pass at roughly one token's worth of weight traffic, the output distribution is provably unchanged, and it pays when draft acceptance is high and the server has spare compute, with 2× to 3× measured on T5-XXL
✗It reliably raises throughput on a fully loaded server, so it should always be enabled. On a batch-saturated GPU the draft and verification flops compete with batching, which is usually the better use of the same hardware.

Why: Speculation is exact by construction. The target model verifies every drafted token and resamples on the first disagreement, so the output distribution is unchanged, which is the part of the Leviathan, Kalman, and Matias result people miss. The speedup comes from verifying k tokens in one weight sweep, and it needs both a well-matched draft and idle compute, which is why it serves latency-sensitive tiers rather than throughput-saturated ones.

6GPU utilization counters are a sound autoscaling signal for an LLM inference fleet.

✗True. A replica can report high utilization while its true limit is KV memory, and can report headroom while queue depth and TTFT are already blowing the SLO, so the counter does not track the binding resource.
✓False

Why: The binding resources are KV cache capacity and memory bandwidth, so the honest signals are the ones the engine exports, KV utilization, queue depth, and the SLOs themselves, with TTFT p95 as the earliest warning. The scaling loop also has to respect GPU physics, since a new replica takes minutes to schedule, pull 16 GB of weights, and warm up, and a draining replica must let its live streams finish, so the fleet scales ahead of the curve instead of chasing a counter.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, The vLLM paper (SOSP 2023). Source of the 20.4 to 38.2 percent KV utilization measurement, the 16-token default block size, the OPT-13B 800 KB per token example, and the 2 to 4x throughput claim over FasterTransformer and Orca.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. Iteration-level scheduling and selective batching, with the 36.9x throughput result against FasterTransformer at the same level of latency.
Leviathan, Kalman, and Matias, Fast Inference from Transformers via Speculative Decoding, The speculative decoding construction, output distribution exactly preserved, 2x to 3x acceleration measured on T5-XXL.
meta-llama/Llama-3.1-8B-Instruct, The config.json behind the KV math, 32 layers, 32 attention heads, 8 KV heads, hidden size 4096, 131,072-token window, bf16.
NVIDIA H100 Tensor Core GPU, The datasheet numbers used throughout, 80 GB of HBM at 3.35 TB/s on the SXM part, with FP16 tensor compute quoted at 1,979 teraFLOPS with sparsity, roughly 990 dense.
vLLM documentation, quantization, The supported quantization surface, AutoAWQ, GPTQModel, FP8 W8A8, INT4 W4A16, INT8 W8A8, and quantized KV cache.
SGLang documentation, RadixAttention and prefix caching, and the production scale claim of trillions of tokens per day across more than 400,000 GPUs.
NVIDIA TensorRT-LLM, NVIDIA's engine, custom kernels for attention, GEMMs, and MoE, in-flight batching, speculative decoding, and prefill-decode disaggregation.