llama.cpp

Part I: The mental model

prompt text
   │
   ▼
llama-cli / llama-server        (tools/)        HTTP, chat templates, slots
   │
   ▼
libllama                        (src/)          tokenize, build graph, sample
   │
   ▼
ggml compute graph              (ggml/)         tensors + ops as explicit DAG
   │
   ▼
backend scheduler               (ggml-backend)  split graph across devices
   │
   ├──► CPU backend             (ggml-cpu/)     SIMD kernels, quantized dot products
   ├──► CUDA backend            (ggml-cuda/)    one .cu file per op family
   └──► Metal / Vulkan / SYCL   (ggml-*/)       same ops, different chips
   │
   ▼
GGUF file, memory-mapped        (gguf)          weights live in page cache

One sentence of identity: llama.cpp is a dependency-free C/C++ stack that loads a single-file model format, expresses a transformer's forward pass as an explicit computation graph, and executes that graph on whatever mix of CPU and GPU hardware you happen to own. Every layer of the diagram is in one repository, which is rare and is the reason the project is such a good read: you can start at a printed character and end at a hand-written AVX2 dot product without ever leaving the tree.

The layers matter because they are also the module boundaries. The tools in tools/ only speak the public API in include/llama.h. The library in src/ knows about vocabularies, KV caches, and model architectures, but does no arithmetic; it only builds graphs. The tensor library in ggml/ knows nothing about language models at all; it executes graphs of generic operations on generic tensors, and its backends compete to implement the same small operation set on every chip that matters. The point of ggml is not to be the fastest tensor library on any one chip but to make the same model file run on nearly every chip, and the rest of this chapter is the story of how that decision plays out.

One recent structural fact is worth absorbing before anything else, because it changed the shape of the code paths described below: llama-cli was rewritten as a thin interactive client. On current master it spawns a local llama-server child process (or connects to an existing one via --server-base) and talks to it over HTTP, so the CLI, the web UI, and your OpenAI-client scripts all exercise exactly the same serving code. The classic single-process loop, tokenize, decode, sample, print, still exists verbatim in tools/completion/ as llama-completion, and it remains the best place to step through the core API in a debugger.

Part II: Using it

Installing

Prebuilt packages exist for every platform. Homebrew covers both macOS and Linux, winget covers Windows, and conda-forge covers all three (with CUDA and Vulkan variants on Linux and Windows):

brew install llama.cpp        # macOS and Linux
winget install llama.cpp      # Windows
conda install -c conda-forge llama-cpp

Building from source takes about a minute because the project has essentially no dependencies beyond a C/C++ toolchain and CMake:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

Binaries land in build/bin/. On Apple Silicon the Metal backend is enabled by default; on Linux add -DGGML_CUDA=ON to the first cmake invocation for NVIDIA GPUs, or the Vulkan, SYCL, or HIP equivalents (-DGGML_VULKAN=ON and so on) for everything else. A CUDA build takes considerably longer than a CPU build because it compiles kernels for several GPU architectures.

First session

The fastest first run uses the -hf flag, which pulls a GGUF file straight from Hugging Face into a local cache and drops you into an interactive chat:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

You will see the model download once, then a chat prompt. Type a message, get streamed tokens back, press Ctrl+C twice to leave. The same flag works for the server, which speaks the OpenAI API on port 8080 by default and ships a built-in web UI:

llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
resp = client.chat.completions.create(
    model="gemma-3-1b-it",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

The flags worth learning first, with their current defaults: -m points at a local GGUF file instead of a Hugging Face repo; -ngl controls how many layers go to the GPU and now defaults to auto, which measures free VRAM and fits as many layers as possible (the old advice of passing 99 still works, spelled -ngl all); -c sets context size and defaults to 0, meaning "use the model's own training context length"; -t sets CPU threads; and --jinja chat templating is on by default. Partial offload remains the distinctive trick: a model that does not fit in VRAM can put some layers on the GPU and run the rest on the CPU, and with auto you no longer even have to compute the split yourself.

Making your own GGUF

The workflow that turns a Hugging Face checkpoint into a local model has three steps: convert, quantize, run.

# 1. convert safetensors to a full-precision GGUF
pip install -r requirements.txt
python convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf

# 2. shrink it (Q4_K_M is the usual sweet spot)
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# 3. run it
llama-cli -m model-Q4_K_M.gguf

Two beginner mistakes cluster here. The first is quantizing from an already-quantized file. llama-quantize will refuse unless you pass --allow-requantize, and it refuses for a reason: requantizing compounds rounding error, so quality is strictly worse than quantizing from the f16 or bf16 original. Wrong and right:

# wrong: Q8_0 -> Q4_K_M loses more than f16 -> Q4_K_M
llama-quantize --allow-requantize model-Q8_0.gguf model-Q4_K_M.gguf Q4_K_M

# right: always quantize from the highest precision you have
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

The second is feeding a chat model raw text without its template. llama-cli applies the model's chat template automatically, but llama-completion and the /completion endpoint do not; an instruction-tuned model given a bare string will often ramble or answer as if autocompleting a document. If you use the raw completion path with a chat model, you must format the prompt with the model's own special tokens yourself, which is exactly the job the template exists to do for you.

Going lower than 4 bits

Below roughly 3 bits per weight you should use an importance matrix, which tells the quantizer which weights the model is most sensitive to. You generate one by running calibration text through the full-precision model, then hand it to the quantizer:

llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.gguf -ngl all
llama-quantize --imatrix imatrix.gguf model-f16.gguf model-IQ2_M.gguf IQ2_M

The imatrix run takes minutes because it is real inference over the calibration corpus. Skipping it for IQ-series quants produces files that run fine but degrade noticeably; the community GGUFs on Hugging Face that people find "surprisingly good at 2 bits" were almost all made with one.

Part III: When it is the right tool

llama.cpp wins whenever the hardware is yours and the user count is small. Laptops, workstations, phones, single-GPU homelabs, air-gapped machines, CPU-only servers, and edge devices are its home turf, and it is the only serious engine where "no GPU at all" is a fully supported configuration rather than a fallback. It also wins on model availability: the GGUF ecosystem means nearly every open model appears in quantized form within days of release, often within hours.

The alternatives divide by deployment. vLLM and SGLang assume datacenter GPUs and optimize aggregate throughput across many concurrent users; if you are serving a product to hundreds of simultaneous requests on A100s or H100s, they are the right tools and llama.cpp is not. MLX is worth a look if you are exclusively on Apple Silicon and want a Python-native research workflow. Ollama and LM Studio are not alternatives so much as packaging: both run llama.cpp (or its ggml lineage) underneath and trade flexibility for convenience.

The architecture-shaped warning: do not put a bare llama-server on the public internet, and do not build a multi-tenant product on the assumption that it will behave like a datacenter serving engine. The server ships with no authentication by default (there is an --api-key flag, but that is a shared secret, not a user system), and its slot-based scheduler is designed for a handful of concurrent users, not thousands. The safe shapes look like this:

safe:      you ──► llama-server on localhost
safe:      team ──► reverse proxy (TLS + auth) ──► llama-server on private net
dangerous: internet ──► llama-server :8080 (no auth, no rate limit)
dangerous: 10k users ──► one llama-server (slot scheduler is not a fleet scheduler)

If traffic outgrows a box, the answer is not a bigger llama-server but a different engine class; my LLM serving design write-up covers where that line sits and why.

Part IV: The full life of one token

This is the core of the chapter. The canonical operation is one token generation step: you have typed a prompt into llama-cli, and we follow the machinery that produces the next character on your screen. Each stage names the actual files involved.

Stage 1: the CLI, which is a client

llama-cli lives in tools/cli/. Its cli_context (in cli-context.cpp) either spawns a local llama-server child, forwarding all the model-relevant flags to it, or connects to an existing server if you passed --server-base. Your typed message is posted as a chat completion request by the code in cli-client.cpp, and streamed tokens come back over HTTP to be printed. This means the "real" execution path for interactive chat is the server path, which is what we follow.

Stage 2: HTTP, tasks, and a slot

Inside tools/server/, server-http.cpp parses the request and server-task.cpp wraps it as a task on the queue in server-queue.cpp. The scheduler heart is server-context.cpp: a free server_slot is claimed by launch_slot_with_task(), and from then on the slot owns this conversation's share of the KV cache. Each iteration of the server loop calls update_slots(), which gathers work from every active slot into one batch, which is how a chat you are having and a request from the web UI can share a single forward pass.

Stage 3: templating and tokenization

The chat messages are rendered into one prompt string by the template code in common/chat.cpp (a Jinja subset engine, on by default). Tokenization is common_tokenize(), a thin wrapper over llama_tokenize(), implemented in src/llama-vocab.cpp. That file contains real implementations of SentencePiece-style and BPE tokenizers plus the per-model pretokenization quirks, which is why a GGUF file needs no external tokenizer: the vocabulary, merges, and special tokens were all baked into the file's metadata at conversion time. For the generation step we care about, the prompt is already in the cache, so the "input" is just the single token sampled last step.

Stage 4: the batch enters llama_decode

update_slots() adds each slot's pending token to a llama_batch via common_batch_add(), tagged with its position and sequence id, then calls llama_decode(). The context object behind that call lives in src/llama-context.cpp, with batch splitting logic in src/llama-batch.cpp: a logical batch of up to n_batch tokens (default 2048) is processed in physical chunks of n_ubatch (default 512). For our single decode token the batch is tiny, and that difference is the whole performance story of decoding: the same graph that saturated the machine during prompt processing now moves one token's worth of activations past billions of weights, so it is memory-bound.

Stage 5: building the ggml graph

llama_decode() does no math. It builds a graph: src/llama-graph.cpp provides the llm_graph_context toolkit of reusable pieces (attention blocks, feed-forward blocks, rope, norms), and src/llama-model.cpp contains a per-architecture build function that assembles them in the right order for LLaMA, Qwen, Gemma, and the dozens of other supported families. The output is a ggml_cgraph, an array of node tensors in topological order, built with ggml_build_forward_expand(). Nothing has executed yet; the graph is a plan.

Stage 6: KV cache views

The graph does not copy the KV cache; it references it. src/llama-kv-cache.cpp (with the cell bookkeeping in llama-kv-cells.h, and sibling files for sliding-window and hybrid variants) owns big per-layer K and V tensors and hands the graph builder two things: a view where this step's new K and V vectors must be written, and views over all occupied cells for attention to read. The cache update is therefore not a separate phase, it is literally nodes in the graph: a cpy into the cache view, sequenced before the attention op that reads it. When the forward pass runs, the cache has been extended as a side effect.

Stage 7: the backend scheduler splits the graph

Now ggml takes over. ggml/src/ggml-backend.cpp implements ggml_backend_sched, which walks the graph and assigns each node to a backend, primarily by asking where the node's weights live: layers offloaded at load time (the -ngl decision) have their tensors in CUDA or Metal buffers, the rest sit in mmapped host memory. The scheduler cuts the graph into contiguous splits per backend, inserts the copies needed to move activations across boundaries, and reserves intermediate memory through the graph allocator in ggml/src/ggml-alloc.c, which reuses buffers whose contents are dead, so the working memory for a whole forward pass is far smaller than the sum of its tensors. Then ggml_backend_sched_graph_compute() runs the splits in order.

Stage 8: kernels, CPU and GPU

On CPU, ggml/src/ggml-cpu/ executes nodes across a thread pool. The hot path is the quantized matrix-vector product: quants.c and the per-ISA code under arch/ (selected through simd-mappings.h) implement dot products that unpack 4-bit blocks and multiply-accumulate them with AVX2, AVX-512, or NEON intrinsics, without ever materializing dequantized weights in memory. On an NVIDIA GPU, ggml/src/ggml-cuda/ provides one file per op family, including dedicated quantized matmul kernels and fused flash-attention kernels; Metal and Vulkan backends mirror the same op set for Apple and everything-else hardware. This is the payoff of the explicit graph: a backend is "just" this operation set, so the same model file ran on your laptop and your GPU server without either knowing.

Stage 9: logits and the sampling chain

The last graph node is the output projection; its result row for our sequence is fetched with llama_get_logits_ith(). Sampling is a chain of composable samplers, implemented in src/llama-sampler.cpp and assembled from user flags by common/sampling.cpp: repetition penalties first, then truncation filters such as top-k, top-p, and min-p, then temperature, then the final random draw. The server calls common_sampler_sample(), gets back a token id, and hands it to process_token(), which checks stop conditions and end-of-generation tokens.

Stage 10: a piece of text, and the loop closes

The token id becomes UTF-8 bytes via the vocabulary (llama_token_to_piece(), again llama-vocab.cpp), is pushed to the HTTP stream by the code in server-stream.cpp, arrives at llama-cli's client loop, and is printed. The sampled token id is also appended to the slot's pending input, so the next update_slots() iteration feeds it back into stage 4. One token took: zero heap allocations of weights, one graph build, one scheduled execution across one or more backends, one cache append, one sample, a few bytes over a socket. That loop runs until a stop token or your Ctrl+C.

Part V: Deep dive: ggml

ggml lives in-tree under ggml/ and is developed in lockstep with llama.cpp. Its core is startlingly small: the API in ggml/include/ggml.h, the implementation in ggml/src/ggml.c, and around them the allocator, the backend abstraction, and the backend implementations.

Tensors and the graph

A ggml_tensor is a struct with up to four dimensions in ne[], byte strides in nb[], a type (f32, f16, or one of the many quantized block types), an op code saying which operation produced it, and src[] pointers to its operands. Tensors are created inside a ggml_context, a bump-allocated arena, so building a graph performs no per-node malloc. Because every tensor remembers its producer, "building the model's forward pass" is just calling functions like ggml_mul_mat() and ggml_soft_max() that create result tensors, then calling ggml_build_forward_expand() on the final one to flatten the dependency DAG into an ordered ggml_cgraph (default capacity GGML_DEFAULT_GRAPH_SIZE, 2048 nodes).

x ──► rms_norm ──► mul(=·w_norm) ──► mul_mat(W_q) ──► rope ──┐
                                 ├─► mul_mat(W_k) ──► rope ──┼─► cpy into KV ──► attn ...
                                 └─► mul_mat(W_v) ───────────┘
       (each arrow = a ggml_tensor node; the cgraph is this DAG, topologically sorted)

The graph is rebuilt for every llama_decode() call, which surprises people who expect "static graph" to mean "built once". Building is cheap (arena allocation, no kernels), and rebuilding is what lets batch size, sequence positions, and cache views change between steps while execution stays planned. A second misconception to retire: ggml is not a training framework with autograd bolted off; there is an optimizer module (ggml-opt.cpp) used for small fine-tuning jobs, but the design center is inference, and the API gives you graphs, not gradients, by default.

Memory planning

ggml-alloc.c implements the graph allocator (ggml_gallocr): given a topologically sorted graph, it walks the nodes, tracks when each intermediate tensor's last consumer has run, and assigns offsets in a shared buffer so that dead tensors' memory is immediately reused, including in-place reuse where an op may overwrite its input. This is why llama.cpp can state its compute buffer size up front at load time, and why that buffer is megabytes when the naive sum of intermediates would be gigabytes. The trade-off is the classic static-planning one: the plan is only valid for graphs of the shape it was measured for, which the scheduler handles by reserving against a worst-case graph.

Backends and the split scheduler

A backend is an implementation of the interface in ggml-backend.h: buffer types (where can tensors live), and graph computation (run these nodes). The registry in ggml-backend-reg.cpp enumerates what was compiled in, and backends can even be loaded as dynamic libraries. The scheduler's split algorithm is the piece worth studying in ggml-backend.cpp: weights pin their nodes to the backend owning their buffer, unpinned nodes are absorbed into neighboring splits to minimize boundary copies, and the resulting plan is what makes -ngl 20 on a 40-layer model genuinely run half the network on GPU and half on CPU with two activation copies per step. The famous trap here: if your build has no GPU backend compiled in, -ngl silently does nothing, and the model runs entirely on CPU; the load log tells you which backends registered and how many layers were offloaded, and reading that log is the first debugging move for any "why is it slow" question.

Part V continued: Deep dive: GGUF and the quantization families

The file format

GGUF is a binary container defined alongside ggml (implementation in ggml/src/gguf.cpp, Python tooling in gguf-py/). The layout:

┌────────────────────────────────────────────────┐
│ magic "GGUF"  │ version (3) │ tensor_count u64 │
│ metadata_kv_count u64                          │
├────────────────────────────────────────────────┤
│ metadata: key → typed value                    │
│   general.architecture = "llama"               │
│   llama.block_count = 32 ...                   │
│   tokenizer.ggml.tokens = [...]                │
├────────────────────────────────────────────────┤
│ tensor infos: name, dims, type, offset         │
├────────────────────────────────────────────────┤
│ tensor data (aligned, default 32 bytes)        │
└────────────────────────────────────────────────┘

Two properties do the work. Self-description: hyperparameters, tokenizer vocabulary, chat template, and quantization details are key-value metadata in the file, so there are no sidecar configs and old binaries can skip keys they do not know. Alignment: the tensor data region is laid out so the whole file can be memory-mapped (src/llama-mmap.cpp) and weights used directly from page cache; "loading" a 40 GB model is mostly the kernel faulting pages in on first touch, and a second process opening the same file pays nothing.

What the bits buy

All llama.cpp quantization is block-wise: small groups of weights share scale factors, and the structs are declared plainly in ggml/src/ggml-common.h. The simplest family uses 32-weight blocks; block_q4_0 is one fp16 scale plus 32 four-bit values, 18 bytes per 32 weights, hence 4.5 bits per weight effective. The K-quants use 256-weight super-blocks with a second level of quantization applied to the scales themselves: block_q4_K holds two fp16 super-scales, twelve bytes of 6-bit sub-block scales and minimums for its eight 32-weight sub-blocks, and 128 bytes of nibbles, 144 bytes per 256 weights, again 4.5 bits per weight but with much finer-grained scaling for the same budget. The mixes you download (Q4_K_M and friends) additionally spend more bits on the tensors where error hurts most, such as attention V and feed-forward down projections.

The IQ series gets below 3 bits by replacing independent rounding with small codebooks of allowed weight patterns, and it is where the importance matrix stops being optional: the quantizer minimizes error weighted by each weight's measured activation statistics, spending its tiny bit budget on the weights the calibration run proved matter. The bet underneath every family is the same: local inference is bound by memory bandwidth, not arithmetic, so every bit shaved off the weights is speed as well as space. The arithmetic (not a benchmark): a 7B model at Q4_K_M is roughly 4 GB of weights, and every generated token must stream essentially all of them, so a machine with 100 GB/s of memory bandwidth cannot exceed roughly 25 tokens per second no matter how fast its ALUs are, while the same model at f16 (14 GB) caps below 8. That is why quantization doubles as the project's performance strategy, and why the kernels dequantize inside the dot product rather than ever writing f16 weights back to memory.

Two corrections to popular beliefs. First, perplexity deltas are a coarse instrument: a quant that adds 1 percent perplexity can still visibly change style or reliability on structured tasks, so evaluate on your task before standardizing on an aggressive quant. Second, quantization here is weights-only: activations and KV cache stay in float by default (the cache can separately be quantized to save memory), so "Q4 model" does not mean the whole computation happens in 4 bits.

Part V continued: Deep dive: the server, slots, and continuous batching

llama-server is the most load-bearing tool in the tree: the OpenAI-compatible API, the web UI, and now the CLI all sit on it. Its concurrency model is slots. A slot is a long-lived serving lane: an assigned share of the context, its own sampling state, and its own cache region, identified by sequence id inside the single shared llama_context. The number of slots comes from --parallel, whose default is now -1, meaning the server chooses automatically.

requests ──► queue (server-queue.cpp)
                 │ launch_slot_with_task()
   ┌─────────┬─────────┬─────────┐
   │ slot 0  │ slot 1  │ slot 2  │   each: seq id, sampler, cache share
   └────┬────┴────┬────┴────┬────┘
        └── update_slots(): one llama_batch with every slot's next tokens
                 │
                 ▼
          llama_decode()  ── one forward pass serves all active slots

This is continuous batching in the same sense vLLM means it, scaled to a workstation: membership in the batch is reconsidered every iteration, a slot that finishes frees its lane immediately, and prompt processing for a new request can share iterations with decoding for old ones, with long prompts chunked by the batch limits so decode latency stays flat. The server also implements prompt caching per slot, so a follow-up message in a chat only processes the new tokens, plus speculative decoding when you supply a draft model, and endpoints for embeddings, reranking, tokenization, health, and Prometheus-style metrics (which expose, among other things, the count of llama_decode() calls and average batch occupancy).

The trap to internalize: slots divide the context, they do not multiply it. The context you configure is shared across serving lanes, so raising parallelism without raising -c gives each conversation a smaller window, and a request that needs more than one slot's share will be rejected or truncated depending on settings. Sizing a server is therefore one multiplication: (context per user) times (concurrent users) must fit in the KV memory your hardware can hold, and the KV cache, not compute, is usually what runs out first.

Part VI: Reading the repository

A staged plan. Every path below exists on master as of July 2026; the project moves fast, so expect drift in the small names but not in the layering.

Stage 0: orientation (one evening)

Read README.md, docs/build.md, and docs/install.md. Build from source and run llama-cli -hf ggml-org/gemma-3-1b-it-GGUF. You should be able to answer: what are the three layers of the tree (tools, libllama, ggml)? What is a GGUF file? What does -ngl do?

Stage 1: the public API and the classic loop

Read include/llama.h top to bottom (it is one file and heavily commented), then tools/completion/completion.cpp, which uses it the classic way, and skim tools/cli/ to see the client-over-server rewrite. Afterwards you should be able to answer: what is the difference between a model and a context? What does llama_decode() take and return? Where do logits live and who owns sampling?

Stage 2: libllama internals

In src/, read in this order: llama-model-loader.cpp (GGUF to tensors), llama-vocab.cpp (tokenizers), llama-context.cpp and llama-batch.cpp (the decode path), llama-graph.cpp plus one architecture's build function in llama-model.cpp, llama-kv-cache.cpp with llama-kv-cells.h, and llama-sampler.cpp. Questions to answer: where exactly does the KV cache get written during a forward pass? Why is the graph rebuilt every call? How does a new model architecture get added?

Stage 3: ggml

Read ggml/include/ggml.h's long header comment, then ggml/src/ggml-alloc.c, ggml/src/ggml-backend.cpp (the sched code), and in ggml/src/ggml-cpu/ the files quants.c and vec.cpp for one ISA. Questions: how does the allocator reuse memory? How does the scheduler decide splits? What does a Q4_K dot product actually do per block?

Stage 4: the format and the quantizers

Read ggml/src/ggml-common.h (the block structs), src/llama-quant.cpp, tools/quantize/, tools/imatrix/, and skim gguf-py/gguf/gguf_writer.py next to convert_hf_to_gguf.py. Questions: why do K-quants quantize their own scales? What does the imatrix change in the optimization objective? What decides which tensors get which types in a _M mix?

Stage 5: the server

Read tools/server/server-context.cpp around launch_slot_with_task(), update_slots(), and process_token(), with server-queue.cpp beside it. Questions: how do slots share one context? Where does continuous batching actually happen? What happens when a prompt exceeds a slot's context share?

Where not to start: the per-op GPU kernel files under ggml/src/ggml-cuda/ (hundreds of files, meaningful only once the graph layer is solid), the architecture zoo in src/models/, vendor/, and the multimodal tree under tools/mtmd/. All are rewarding later and disorienting first.

Part VII: Hands-on labs

Each lab teaches one concept from the deep dives. Commands assume a source build with binaries on your PATH; outputs vary by hardware, so treat the numbers as shapes, not targets.

Lab 1: bandwidth is destiny. Run the built-in benchmark on one model at two quantization levels:

llama-bench -m model-Q4_K_M.gguf -m model-Q8_0.gguf

Observe the pp (prompt processing) and tg (token generation) columns. Generation speed should scale close to inversely with file size while prompt speed changes much less, because generation streams the weights per token (bandwidth-bound) and prompt processing amortizes them over many tokens (compute-bound). This is the Part V bandwidth argument made measurable.

Lab 2: convert and quantize your own model. Take a small Hugging Face checkpoint (a 0.5B or 1B model keeps this fast), run convert_hf_to_gguf.py, then produce Q8_0, Q4_K_M, and Q2_K files and compare sizes with ls -lh. Open the file metadata with:

python gguf-py/gguf/scripts/gguf_dump.py model-Q4_K_M.gguf | head -50

(path may drift; any GGUF dump tool works). Observe the key-value metadata from Part V's format diagram, and that different tensors in the same "Q4" file carry different types.

Lab 3: measure what quantization costs. Run perplexity over a small text file for two quants of the same model:

llama-perplexity -m model-Q8_0.gguf -f wiki.test.raw
llama-perplexity -m model-Q2_K.gguf -f wiki.test.raw

Expect the Q2_K perplexity to be visibly worse and Q8_0 to be nearly indistinguishable from f16. Then build an imatrix with llama-imatrix and requantize the low-bit file with it, and measure again: the gap should narrow. This is the imatrix objective from the deep dive doing its job.

Lab 4: watch the scheduler split a graph. On a machine with a GPU backend, load a model three ways:

llama-cli -m model.gguf -ngl 0 -p "hi" -n 8
llama-cli -m model.gguf -ngl 10 -p "hi" -n 8
llama-cli -m model.gguf -ngl all -p "hi" -n 8

Read the load logs each time: which backends registered, how many layers were offloaded, and the compute buffer sizes per backend. With partial offload you are seeing the ggml-backend split scheduler from Part V configured before your eyes, and the tokens-per-second difference between the three runs is the price of the CPU-resident layers.

Lab 5: slots under concurrency. Start a server with explicit parallelism and a fixed context, then fire simultaneous requests:

llama-server -m model.gguf -c 8192 --parallel 4 --metrics
for i in 1 2 3 4; do
  curl -s http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Count to 30 slowly."}]}' & done; wait
curl -s http://localhost:8080/metrics | grep -i slot

Observe in the server log that four slots activate and decode interleaves, and in the metrics that batch occupancy per llama_decode() rose above one. Then send a prompt far larger than 2048 tokens (8192 divided by 4 slots) and observe the failure mode from the "slots divide the context" trap.

Lab 6: one token in the debugger. Build with debug info (cmake -B build -DCMAKE_BUILD_TYPE=Debug) and run the classic loop:

gdb --args ./build/bin/llama-completion -m model.gguf -p "The capital of France is" -n 4
(gdb) break llama_decode
(gdb) run

At each stop, look at the batch size (first stop: your prompt tokens; later stops: exactly one token), then step into the graph build and print a few nodes of the ggml_cgraph. This is Part IV made tangible; the moment the batch collapses from N tokens to 1 is the prefill-to-decode transition.

Part VIII: Questions and model answers

1. What is llama.cpp in one sentence? A dependency-free C/C++ inference stack that runs quantized transformer models from a single memory-mappable file on nearly any CPU or GPU, built on the ggml tensor library.

2. What is the relationship between llama.cpp and ggml? ggml is the tensor library underneath: tensors, an explicit compute graph, an allocator, and pluggable backends, developed in-tree under ggml/. libllama builds LLM-shaped graphs (tokenizers, KV cache, architectures) and hands them to ggml to execute; the tools sit on libllama's public header.

3. Why does an explicit static graph matter? Because a backend only needs to implement a fixed operation set to run every model, and a scheduler that can see the whole graph can split it across devices and plan memory reuse ahead of execution. Portability and partial offload both fall out of that one decision.

4. Is the graph built once and reused? No; it is rebuilt on every llama_decode() call. Building is cheap arena allocation with no kernel launches, and rebuilding lets batch contents, positions, and cache views change each step while execution stays fully planned.

5. Walk through one generated token at a high level. The previous token enters a batch in update_slots(); llama_decode() builds a ggml graph that includes writes into the KV cache views; the backend scheduler splits the graph between CPU and GPU by weight residency and executes it; logits come back, the sampler chain picks a token id, the vocabulary turns it into text, and the id is fed back as the next input.

6. What makes GGUF loading fast? The file is designed for mmap: aligned tensor data used directly from page cache, so load time is page faults rather than copies, and self-describing metadata, so no external configs are parsed. A warm second load or a second process is nearly free.

7. What is block-wise quantization and why blocks? Weights are grouped (32 or 256 at a time) and each group shares scale factors, so the format adapts to local weight magnitude at a cost of a fraction of a bit per weight. Per-tensor scaling would lose too much precision; per-weight scaling would cost too many bits; blocks are the workable middle.

8. What distinguishes Q4_K from Q4_0 at the same 4.5 bits per weight? Q4_0 spends its overhead on one fp16 scale per 32 weights. Q4_K uses a 256-weight super-block whose eight sub-block scales and minimums are themselves quantized to 6 bits under two fp16 super-scales, buying finer-grained scaling from the same bit budget, which measurably lowers error.

9. What does an importance matrix change? It reweights the quantizer's error objective using activation statistics gathered by running calibration text through the model, so the scarce bit budget protects the weights that most affect outputs. It matters increasingly as bit rates fall and is effectively required below 3 bits.

10. Why is decode speed bandwidth-bound, and what follows from it? Each generated token must read essentially all weights while doing comparatively little arithmetic, so tokens per second is approximately memory bandwidth divided by model bytes. It follows that halving model size roughly doubles generation speed, which is why quantization is the performance strategy and not just a memory strategy.

11. How does partial GPU offload work? At load, -ngl (now defaulting to auto-fit) decides which layers' weights go into GPU buffers. The backend scheduler then assigns graph nodes to backends by weight residency, cuts the graph into per-backend splits, and inserts activation copies at the boundaries, so one forward pass runs partly on each device.

12. How does llama-server handle concurrency? Slots: each active request owns a sequence id, sampler state, and a share of the shared context, and every iteration update_slots() rebuilds one batch from all active slots for a single llama_decode(). That is continuous batching at workstation scale, with prompt caching per slot on top.

13. When should you reach for vLLM instead? When the deployment is datacenter-shaped: many concurrent users, dedicated server GPUs, throughput as the objective. vLLM's paged KV cache and iteration-level scheduler are built for that regime; llama.cpp optimizes reach and footprint for one machine and a handful of users, and neither substitutes for the other.

14. A user reports the model outputs template tags like <|im_start|> or answers as if continuing a document. What happened? The chat template was not applied, typically because they used the raw completion endpoint or llama-completion with a chat-tuned model, or because template detection failed for a new model. Fix by using the chat endpoint or CLI (templating is on by default) or supplying the template explicitly.

15. Generation is much slower than a friend's identical GPU. First three checks? Check the load log for which backends registered (a CPU-only build ignores -ngl silently), how many layers were actually offloaded (VRAM pressure can shrink the auto fit), and whether the context or KV cache spilled layers to CPU. The answer is almost always in the startup log, not the generation code.

16. Why does requantizing a quantized model degrade quality? Quantization is lossy rounding against the original weights; quantizing already-rounded values compounds two rounding errors and can shift block scales chosen for the first grid. That is why llama-quantize demands --allow-requantize to do it at all, and why serious quants start from f16 or bf16 conversions.

Part IX: Design lessons

Optimize for reach, then let reach compound. ggml trades peak per-chip performance for running everywhere, and the payoff was an ecosystem: because every device could run GGUF, every model got converted, and because every model was available, every tool built on llama.cpp. The same dynamic built SQLite and ffmpeg; ubiquity is a moat that benchmarks do not measure.

A single-file, self-describing artifact is a superpower. GGUF holds weights, tokenizer, and configuration in one mmappable file with forward-compatible key-value metadata. Every format that wins distribution works this way, from SQLite databases to container images: the artifact carries its own interpretation, so tooling composes without coordination.

Make the expensive thing explicit. The build-graph-then-execute split looks ceremonial next to eager frameworks, but it is what makes memory planning, backend splitting, and portability tractable. Compilers, query planners, and render graphs in game engines all rediscover the same shape: declare the whole computation first, and optimization becomes a pass instead of a heuristic.

Co-locate the format with the kernels that read it. Quantization formats in llama.cpp were designed with their dot products, so data layout and SIMD access patterns evolved together, and dequantization never round-trips through memory. Columnar databases (Parquet plus vectorized readers) teach the identical lesson: a storage format is only as good as the inner loop it was shaped for.

Collapse code paths onto one battle-tested one. Rewriting llama-cli as a client of llama-server meant chat, web UI, and API traffic now exercise identical serving code, so a fix in slot handling fixes every frontend. This is the same argument as CLIs that call their own public HTTP APIs; second implementations of the same behavior are where bugs live.

Part X: Memorization framework

One sentence to keep: llama.cpp runs a memory-mapped, block-wise quantized model file through an explicitly built ggml graph that a scheduler splits across CPU and GPU backends, one bandwidth-bound token at a time.

Text → Token → Slot → Batch → Graph → Split → Kernel → KV → Logits → Sample → Piece

The chain mapped to real files:

Text    tools/cli/, tools/server/server-http.cpp
Token   src/llama-vocab.cpp, common/chat.cpp
Slot    tools/server/server-context.cpp
Batch   src/llama-batch.cpp, src/llama-context.cpp
Graph   src/llama-graph.cpp, src/llama-model.cpp
Split   ggml/src/ggml-backend.cpp, ggml/src/ggml-alloc.c
Kernel  ggml/src/ggml-cpu/, ggml/src/ggml-cuda/
KV      src/llama-kv-cache.cpp
Logits  llama_get_logits_ith (include/llama.h)
Sample  src/llama-sampler.cpp, common/sampling.cpp
Piece   src/llama-vocab.cpp, back over HTTP

Memorize these:

• Three layers, one repo: tools → libllama (src/) → ggml. Tools see only llama.h.
• Q4_0: 32 weights, fp16 scale, 18 bytes, 4.5 bpw. Q4_K: 256-weight super-block, quantized 6-bit sub-scales, 144 bytes, 4.5 bpw spent smarter.
• Decode tokens/s ≈ memory bandwidth ÷ weight bytes; quantization is a speed feature.
• The graph is rebuilt every decode call; the KV cache update is nodes inside it.
• Slots divide the context; continuous batching happens in update_slots() around one shared llama_decode().

Part XI: Papers and further reading

The ideas in this chapter trace back to a handful of papers and a few load-bearing pull requests, and each one rewards a direct read. Where this site covers the same ground in depth, the companion link points there.

1. Touvron et al., LLaMA, Open and Efficient Foundation Language Models, 2023. The model the project began as a weekend port of, and the architecture whose forward pass Part IV builds as a graph. The language models from scratch class on this site constructs the same architecture by hand.
2. Philpax and the ggml contributors, The GGUF file format specification, 2023. The normative description of the single-file container from Part V, the key-value metadata and the alignment rules that make mmap loading work.
3. Kawrakow, k-quants, llama.cpp pull request 1684, 2023. Where the 256-weight super-blocks with quantized sub-scales were introduced, along with the perplexity tables that justified each mix.
4. Kawrakow, Importance Matrix calculation, llama.cpp pull request 4861, 2024. The imatrix mechanism from Part II, activation statistics gathered from calibration text that reweight the quantizer's error objective.
5. Frantar et al., GPTQ, Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022. The second-order one-shot quantizer that set the pace of the wider low-bit race llama.cpp's simpler block formats compete in.
6. Lin et al., AWQ, Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023. Protecting the few weights that activations prove salient, the same instinct the importance matrix brings to this repo's quantizers.
7. Dettmers and Zettlemoyer, The case for 4-bit precision, k-bit Inference Scaling Laws, 2022. Evidence across model families that roughly four bits per weight is the accuracy-per-byte sweet spot, which is why Q4_K_M is the default advice.
8. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022. The draft-model acceleration llama-server offers, exact sampling preserved while a small model proposes tokens for the big one to verify.
9. Ma et al., The Era of 1-bit LLMs, All Large Language Models are in 1.58 Bits, 2024. The natively ternary training recipe that the TQ1_0 and TQ2_0 quant types exist to serve.
10. Dao et al., FlashAttention, Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. The fused attention idea behind the CUDA backend's flash-attention kernels, covered in the FlashAttention walkthrough.
11. Yu et al., Orca, A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. The origin of iteration-level continuous batching, the idea update_slots() applies at workstation scale.
12. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The datacenter serving engine on the other side of Part III's dividing line, covered in the vLLM walkthrough.

Part XII: Final takeaway