Ollama

Part I: The mental model

ollama run llama3.2 "..."      cmd/cmd.go        cobra CLI, one binary
      |
      v
api client  --HTTP-->  server   api/client.go talks to 127.0.0.1:11434
      |
      v
server/routes.go                gin router: /api/chat /api/generate /api/embed /v1/*
      |
      v
Scheduler   server/sched.go     is this model loaded? estimate VRAM, evict, keep-alive
      |
      v
LlamaServer  llm/server.go      spawn a subprocess:  ollama runner --model ...
      |
      v
runner subprocess               loads the GGUF, serves /completion on a private port
      |  cgo
      v
llama.cpp / GGML  (llama/)      tokenize -> decode loop -> sample -> detokenize
      |
      v
tokens stream back up           runner -> server -> api client -> your terminal

The one-sentence identity: Ollama is a small, opinionated Go server that turns a model name into a running local LLM, by packaging models like container images, pulling them from a content-addressable registry, and supervising a llama.cpp/GGML runner subprocess through a memory-aware scheduler behind a plain REST API. Everything friendly about Ollama is a thin layer over that idea. You never see a Python environment, a weights file path, a GPU-offload flag, or a tokenizer. You type a name, and a daemon downloads the right artifacts, works out how much of the model fits in your VRAM, launches the engine, and streams tokens back.

Two load-bearing ideas carry the whole design. First, models are packaged and distributed like Docker images. A Modelfile is compiled into a set of layers, each a content-addressable blob keyed by its SHA-256 digest, gathered under a manifest, and served over the standard OCI/Docker registry protocol. Pulling llama3.2 is pulling a manifest and its blobs, and two models that share a base share that base's bytes on disk. Second, inference runs in a supervised subprocess, not in the server process. The server never links the model into its own address space. Instead a scheduler decides when to load a model, estimates how many layers fit on each GPU, spawns a separate ollama runner process that owns the weights, proxies completion requests to it, and unloads it after an idle timeout. That is the keep-alive mechanism, and it is why the second ollama run of the day is instant and the first one after lunch pauses to reload.

Keep those two ideas in view and the repository stops being a maze. The parser, server/images.go, and server/download.go exist to serve the packaging idea. The server/sched.go, llm/, and runner packages exist to serve the subprocess idea. The api, server/routes.go, and openai packages are the front door over both. Everything here is described against the project as it stands in 2026. Ollama moves fast and has been migrating from a pure llama.cpp backend toward its own Go-native inference engine, so where an exact path is likely to have shifted I name the component by its role and say so.

Part II: Using it

Ollama ships as a single self-contained binary for macOS, Linux, and Windows, with the GPU backends bundled. The install script is the usual one-liner on Linux, and there are native app installers for macOS and Windows.

# linux
curl -fsSL https://ollama.com/install.sh | sh

# then, in one command, pull and chat with a small model
ollama run llama3.2

# non-interactive: prompt on the command line, answer to stdout
ollama run llama3.2 "explain a bloom filter in two sentences"

The first run of a model you do not have pulls it first, then drops you into an interactive REPL. Under the hood ollama run needs a server. On the desktop apps the server is already running as a background service. From a raw binary you start it yourself, and it listens on 127.0.0.1:11434 by default.

# start the daemon (the desktop apps do this for you)
ollama serve

# in another shell, the everyday commands
ollama pull nomic-embed-text     # download a model without running it
ollama list                      # what is on disk
ollama ps                        # what is loaded in memory right now
ollama show llama3.2             # the Modelfile, params, template, and license
ollama rm  llama3.2              # delete a model from disk
ollama cp  llama3.2 my-llama     # copy under a new name

The daemon is the whole product. Every CLI verb above is a thin wrapper that makes an HTTP call to it. Once it is running you can drive it directly, and this is where Ollama earns its place in an application. The native chat endpoint streams newline-delimited JSON objects.

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "name three sort algorithms" }
  ],
  "stream": true
}'

The genuinely useful trick is that the same daemon speaks the OpenAI wire format under /v1. Any code written for the OpenAI SDK runs against a local model by changing the base URL and passing any non-empty API key.

from openai import OpenAI

# point the standard client at the local Ollama server
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "one line: what is keep-alive?"}],
)
print(resp.choices[0].message.content)

There is also a first-party Python and JavaScript client that speaks the native API, which exposes things the OpenAI shape does not, such as raw prompts, per-request model options, and the keep_alive control.

import ollama

resp = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "hello"}],
    options={"temperature": 0.2, "num_ctx": 8192},
    keep_alive="30m",          # keep the model resident for half an hour
)
print(resp["message"]["content"])

Embeddings are a first-class endpoint, which is what makes Ollama a reasonable local backend for retrieval and semantic search. The newer /api/embed takes a single string or a batch and returns a list of vectors.

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["the quick brown fox", "lorem ipsum dolor"]
}'
# -> { "embeddings": [[0.01, -0.02, ...], [...]], "model": "nomic-embed-text", ... }

To make a model your own, write a Modelfile. It is deliberately Dockerfile-shaped. FROM names a base, PARAMETER sets default runtime options, SYSTEM bakes in a system prompt, and TEMPLATE defines how messages are rendered into the prompt using Go template syntax.

# Modelfile
FROM llama3.2

PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER stop "<|eot_id|>"

SYSTEM """
You are a terse assistant. Answer in a single sentence.
"""

ollama create terse-llama -f Modelfile      # compiles the Modelfile into layers
ollama run    terse-llama "what is a raft quorum?"

Now the mistakes newcomers make. First, expecting the run command and the serve daemon to be separate programs. They are the same binary, and every command needs the daemon reachable at OLLAMA_HOST. Second, over-reading the model tag. llama3.2 is llama3.2:latest, which is a specific quantization the registry chose for you, usually a 4-bit one. If you want a particular size or precision, name the tag, for example llama3.1:8b-instruct-q8_0. Third, assuming num_ctx is free. The context length you request sizes the KV cache, which competes with the weights for VRAM, so a large context can push layers off the GPU and onto the CPU and quietly halve your throughput. Fourth, forgetting that the model is resident. Ollama keeps a model loaded for five minutes after the last request by default, so it holds VRAM long after your script exits, which surprises everyone the first time ollama ps shows a model still parked in memory. That behavior is a feature, and the next parts explain exactly how it works.

Part III: When it is the right tool

Ollama is the right tool when you want a local model running with the least possible ceremony. A developer prototyping against an LLM without an API key, a desktop or CLI app that wants to embed a model without asking users to manage weights, a privacy-sensitive workflow that must keep text on the machine, and anyone who wants an OpenAI-shaped endpoint they fully control. Its whole reason to exist is that it hides the parts of local inference that are genuinely annoying, namely finding weights, converting and quantizing them, choosing GPU offload, and wiring up a server.

The honest cases for alternatives start with the engine Ollama sits on. llama.cpp itself, with llama-server and llama-cli, gives you every knob directly and the newest backend features first, at the cost of managing GGUF files and flags by hand. Reach for it when you need control Ollama abstracts away. For serving many concurrent users at high throughput on a GPU box, vLLM, SGLang, and TensorRT-LLM are built for that regime with PagedAttention and continuous batching, and they outclass Ollama on aggregate tokens per second under load. Ollama is tuned for the single-user or few-user local case, not for saturating an H100. Among the local-first tools, LM Studio offers a polished GUI over a similar llama.cpp core, and Jan and GPT4All occupy the same desktop niche. Ollama's distinguishing bet is the registry and Modelfile packaging model plus a clean API, which is why so many local apps target its endpoint specifically.

The architecture-shaped warning is about memory and the CPU cliff. Local inference is bound by memory bandwidth, and the single biggest performance decision is how many transformer layers live on the GPU versus the CPU. When a model plus its KV cache fits in VRAM, generation runs at the GPU's memory bandwidth. When it does not, the layers that spill to system RAM run at an order of magnitude less bandwidth, and a run that was fast becomes a crawl with no error at all.

fits in VRAM:     all N layers on GPU     ->  fast, GPU memory bandwidth
partial offload:  K of N on GPU, N-K CPU ->  each token waits on the CPU layers
                  the model still "works", just many times slower

lever:  smaller quant (q4 vs q8), smaller num_ctx, or a smaller model
        all three free up VRAM so more layers stay on the GPU

Ollama estimates the split for you and prints it, but it cannot invent VRAM. The three levers that keep layers on the GPU are a smaller quantization, a smaller context window, and a smaller model, and knowing that turns a mysteriously slow local model into a solvable problem. This is the local mirror of the interconnect warning in the torchtitan chapter, where the wrong mapping of work to hardware is slow rather than wrong.

Part IV: The full life of one request

The specimen: ollama run llama3.2 "name three sort algorithms" on a machine where the model is already pulled and the daemon is running. Most of the machinery below is the same whether the caller is the CLI, a curl, or the OpenAI client, because they all converge on the same handler.

Stage 1: the cmd CLI

The binary's main hands off to the cmd package, which is a cobra command tree. The run command in cmd/cmd.go builds an api.Client from OLLAMA_HOST and does three things in order. It checks that the model exists locally by calling ShowHandler, and if it does not, it streams a pull first. It sends a load request, which is a generate or chat call with an empty prompt whose only job is to make the scheduler resident the model. Then it enters the interactive loop, or, given a prompt argument, sends exactly one chat request and prints the streamed reply. Everything the CLI does is an HTTP call. There is no in-process inference in the run command at all, which is the first sign that the daemon is the real program.

Stage 2: the api client and the wire

api/client.go is a small typed HTTP client, and api/types.go is the shared vocabulary of the whole system: GenerateRequest, ChatRequest, Message, Options, EmbedRequest, and their response types. The client POSTs the ChatRequest as JSON to /api/chat and, because stream defaults to true, reads the response body as a sequence of newline delimited JSON objects, invoking a callback for each. This request and response vocabulary is worth internalizing early, because the CLI, the Go client, the Python and JS libraries, and the OpenAI shim all ultimately construct these same structs.

Stage 3: server/routes.go dispatches

The daemon's HTTP surface is built in server/routes.go, where a method on the Server constructs a gin router and registers every route. The native API lives under /api with handlers such as ChatHandler, GenerateHandler, EmbedHandler, PullHandler, CreateHandler, ListHandler, ShowHandler, PsHandler, and DeleteHandler. The OpenAI-compatible routes live under /v1 and pass through translation middleware from the openai package before reaching the same handlers. Our request lands in ChatHandler, which validates the body, resolves the model name to a local manifest, and assembles the model's baked-in SYSTEM, TEMPLATE, and default PARAMETER values with the request's own messages and options. Then, crucially, it does not run the model. It asks the scheduler for a runner.

Stage 4: the scheduler decides

server/sched.go is the heart of Ollama's runtime. The handler calls into the scheduler with the model, the merged options (context length matters here because it sizes memory), and the keep-alive duration, and receives back a reference to a running runner over a channel. Inside, the scheduler asks one question. Is a runner for this exact model and configuration already loaded? If yes, it returns the existing reference and increments a refcount. If no, it must load one, and loading is where the memory-aware machinery runs, described in the deep dive below. It estimates VRAM, decides GPU-layer offload, evicts another model if there is not enough room and policy allows, then spawns the runner and waits for it to report healthy. The handler blocks on that channel, so from the caller's point of view a chat request against a cold model simply takes a few seconds longer while the daemon quietly loads it.

Stage 5: the runner subprocess and the backend

The loaded runner is a separate operating-system process. The llm package, in what is effectively llm/server.go, builds a command line that re-invokes the same ollama binary as a hidden runner subcommand, pointed at the model's GGUF blob, told how many layers to place on the GPU, the context size, the parallelism, and the port to listen on. That subprocess loads the weights through cgo bindings to llama.cpp and GGML and serves a small private HTTP API, with a /health endpoint the scheduler polls until the model is ready and a /completion endpoint for generation. The parent server proxies the actual prompt to that private port. This isolation is deliberate. A crash in native inference code takes down a child process rather than the daemon, the weights never sit in the server's heap, and multiple models are just multiple child processes.

Stage 6: tokenize, decode, sample, stream

Inside the runner the request becomes the classic autoregressive loop. The prompt is tokenized, the prefill pass populates the KV cache, and then each step runs a forward pass over the resident layers, samples the next token under the temperature, top-k, top-p, and repeat-penalty settings carried in the options, appends it to the KV cache, and detokenizes the piece back into text. Each piece is written to the runner's HTTP response as it is produced. If a JSON schema or format was requested, a grammar constrains sampling so the output is guaranteed to parse. The loop stops on an end-of-sequence token, a configured stop string, or the predicted-token budget.

Stage 7: back up the stack, then keep-alive

Each generated piece travels back up the same path in reverse. The runner streams it to the server, ChatHandler wraps it in a ChatResponse object and writes it to the client as one line of JSON, the api client's callback fires, and the CLI prints it. When generation ends, the final response object carries done: true along with timing and token-count metrics. Now the important part. The handler releases its reference to the runner, which drops the refcount to zero. The scheduler does not unload immediately. It arms an expiry timer for the keep-alive duration, five minutes by default. If another request arrives first, the timer is cancelled and the model stays hot. If the timer fires, the scheduler kills the runner subprocess and its VRAM is freed. That closes the loop of one request. A name in, weights loaded once and reused, tokens streamed out, and a model left warm for the next call.

Part V: Internals deep dives

Deep dive: the Modelfile and content-addressable storage

A model in Ollama is not a file. It is a manifest that references a set of layers, exactly like a container image, and the Modelfile is its build recipe. The parser package reads a Modelfile into a list of commands, and CreateModel in server/images.go turns those commands into layers. Each instruction becomes its own blob with its own media type.

Every blob is named by the SHA-256 digest of its own bytes and stored under the models directory as blobs/sha256-<hex>, with a dash rather than a colon so it is a valid filename. A manifest, stored under manifests/<registry>/<namespace>/<name>/<tag>, is a Docker v2 manifest JSON that lists the layer digests and media types plus a config blob. The default fully qualified name of llama3.2 is registry.ollama.ai/library/llama3.2:latest, and the library namespace is Ollama's curated catalog.

~/.ollama/models/
  manifests/registry.ollama.ai/library/llama3.2/latest   (JSON manifest)
      |  references by digest
      v
  blobs/
    sha256-a1b2...   image.model     (the GGUF weights, the big one)
    sha256-c3d4...   image.template  (Go template)
    sha256-e5f6...   image.params    (default options)
    sha256-0718...   image.license

two models FROM the same base share the sha256-a1b2 blob on disk

The payoff of content addressing is deduplication and integrity for free. Because a blob's name is its content hash, two models built on the same base weights point at the same physical blob, a corrupted download is detectable by rehashing, and re-pulling a model you already have is a no-op after the manifest check. The trap for newcomers is expecting a portable single file. To move a model you export and re-import through the registry or a Modelfile, not by copying one path, and the image.model blob is a standard GGUF that llama.cpp can read directly if you find it by digest.

Deep dive: pull mechanics and the registry protocol

Because models are OCI-style images, pulling one is speaking the standard Docker registry v2 protocol, and Ollama did not have to invent a distribution format. PullHandler resolves the name, then server/download.go does the work. First a GET of the manifest by tag, then, for each layer the manifest lists that is not already present as a blob, a GET of that blob by digest. Downloads are chunked and run several parts in parallel, they use HTTP range requests so an interrupted pull resumes rather than restarts, and each completed blob is verified against its digest before it is committed into the blob store. The progress bars you watch during ollama pull are per-blob byte counters.

ollama pull llama3.2
  GET /v2/library/llama3.2/manifests/latest        -> manifest JSON (list of digests)
  for each digest not already in blobs/:
    GET /v2/library/llama3.2/blobs/sha256-a1b2...   -> chunked, ranged, resumable
    verify sha256(bytes) == digest                  -> then commit to blobs/
  write the manifest last                           -> the model now "exists"

The manifest is written last on purpose, so a half-finished pull never presents as an installed model. Pushing your own model with ollama push is the mirror image against the same protocol, uploading blobs the registry does not already have and then the manifest, which is why sharing a fine-tune is cheap when it shares a base. The subtlety to know is the tag. A tag is mutable and points at whatever manifest was published for it, while a digest is immutable, so pinning llama3.1:8b-instruct-q4_0 or an explicit digest is how you get reproducibility rather than the moving latest.

Deep dive: the scheduler, memory estimation, and keep-alive

server/sched.go is the piece of Ollama most worth reading, because it is where the product's hardest promises live. Load the right model, on the right devices, without running out of memory, and get out of the way when idle. The scheduler runs a loop over channels of pending requests, finished requests, and expired runners, and it owns the set of currently loaded runners.

When a request needs a model that is not loaded, the scheduler must decide how to fit it. GPU discovery, in the discover package, reports how much free VRAM each device has across CUDA, ROCm, and Metal. Memory estimation, in what is effectively llm/memory.go, reads the GGUF metadata for the model, the per-layer weight sizes, the number of layers, and the size of the KV cache implied by the requested context length and parallelism, then computes how many layers fit on the available GPUs with room for the compute graph and some overhead. The answer is a number of layers to offload, and the rest run on the CPU. If even with zero GPU layers there is not enough system memory, or if another model must move to make room, the scheduler consults its limits. OLLAMA_MAX_LOADED_MODELS caps how many models sit resident at once, and if a new load needs space it evicts the least recently used runner whose refcount is zero. Only then does it spawn the runner and wait on its health check.

Concurrency is the other half. OLLAMA_NUM_PARALLEL controls how many requests one loaded model serves at the same time by giving the runner that many KV-cache slots, and OLLAMA_MAX_QUEUE bounds how many requests wait when everything is busy. These are set to sensible automatic defaults based on available memory, and turning them up trades VRAM for throughput.

Keep-alive is the elegant part. Each runner carries a session duration and an expiry timer. A request increments the runner's refcount, and finishing decrements it. When the count reaches zero, the timer is armed for the keep-alive duration. The keep_alive field on any request, or the global OLLAMA_KEEP_ALIVE, sets that duration, and the semantics are worth memorizing.

keep_alive value        effect
  "5m", "1h", 300        stay resident that long after the last request (default 5m)
  0                      unload immediately when the request finishes
  -1, "-1m", negative    stay resident forever, until evicted or the daemon stops

The whole feel of Ollama, instant on the second call, reload after idle, one warm model at a time on a laptop, is the emergent behavior of this refcount plus timer plus eviction policy. The classic confusion to correct is thinking a model is loaded for the lifetime of a request. It is loaded lazily on first use and unloaded lazily after idle, decoupled from any single call, which is exactly why ollama ps exists to show you what is parked in memory and why it has a UNTIL column.

Deep dive: the runner and the backend, llama.cpp and the new engine

The runner is where Ollama meets the actual math, and it is the part of the codebase in the most active motion, so hold the roles more tightly than the paths. Historically Ollama vendored llama.cpp and GGML under the llama directory and drove them through cgo, and the runner subprocess was a Go program wrapping that C++ engine. The GGML tensor library provides the operators and the GGUF loader, llama.cpp provides the transformer graph, the sampler, and the partial-offload machinery, and the Go runner exposes them over the private /completion, /embedding, and /health HTTP API the scheduler talks to. The GGUF format and block-wise quantization, which is what lets a 4-bit model fit on a laptop, are covered in depth on the llama.cpp page and are exactly what Ollama inherits.

More recently Ollama has been building its own inference engine in Go, with a backend abstraction in the ml package that still uses GGML for the heavy operators through the same GGUF weights, and model architectures reimplemented in Go under a model package. The practical shape is two runner implementations behind one ollama runner command, one wrapping llama.cpp and one driving the native engine, with the server choosing based on whether the model's architecture is supported by the new path. This is how newer capabilities, such as certain vision models, arrive on the native engine while the long tail of architectures keeps running through llama.cpp. Read this area for the boundary rather than the file names. The server hands a runner a GGUF and an offload plan and speaks HTTP to it, and whether the tokens are produced by C++ or by Go on the far side of that boundary is an implementation detail the scheduler does not care about.

The reason the subprocess boundary is drawn exactly there is worth stating plainly. Native inference code, whether C++ or cgo, can segfault or exhaust device memory, and putting it in a child process means a bad model or a bad prompt kills one runner rather than the daemon that other clients depend on. It also lets the same daemon hold several models as several processes, each with its own GGML context, and unload any one by simply killing it, which is a far cleaner memory story than trying to free a model inside a long-lived server heap.

Deep dive: the API surface, native, OpenAI, and embeddings

The native API in server/routes.go is small and regular. /api/generate is single-turn completion over a raw prompt, giving you full control of the text including an optional raw mode that skips templating. /api/chat is the multi-turn endpoint that applies the model's template to a list of role-tagged messages and supports tool calls and image inputs for multimodal models. Both stream newline-delimited JSON by default and accept an options object, which is the runtime knob set, temperature, top_k, top_p, num_ctx, num_predict, stop, seed, repeat_penalty, and num_gpu to force a layer count, among others.

Embeddings have two endpoints for historical reasons. The legacy /api/embeddings takes a single prompt and returns one embedding, while the newer /api/embed takes an input that may be a string or an array and returns a batch of embeddings. Prefer the batch form for indexing a corpus, and pair a dedicated embedding model such as nomic-embed-text or mxbai-embed-large with it rather than a chat model.

The OpenAI compatibility layer, in the openai package, is a translation shim rather than a second engine. It registers /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models, and middleware rewrites the OpenAI request body into Ollama's native ChatRequest or EmbedRequest, calls the same handler, and rewrites the response, including the streamed chunks, back into the OpenAI server-sent-events shape.

POST /v1/chat/completions
   openai middleware:  OpenAI body  --translate-->  api.ChatRequest
   -> ChatHandler (the exact same handler as /api/chat)
   -> scheduler -> runner -> tokens
   openai middleware:  ChatResponse chunks  --translate-->  OpenAI SSE "data:" chunks

Because the OpenAI routes are a thin adapter over the native handlers, everything the daemon can do is reachable both ways, and the compatibility is real rather than a partial reimplementation. The one thing to keep straight is that OpenAI-specific concepts with no local meaning, such as an account's model catalog or server-side tool execution, are mapped as best they can be onto local semantics, so read the OpenAI-compat notes when a field behaves unexpectedly.

Part VI: Reading the repository

The tree is a manageable Go codebase, and the two load-bearing ideas from Part I tell you what each package is for. Where a path is likely to have shifted with the engine migration I say so.

Stage 0, orientation. Read the root main.go and then cmd/cmd.go. Questions. What does the run command actually do before it prints a token, where does OLLAMA_HOST come from, and which commands are pure HTTP calls to the daemon, which is nearly all of them.

Stage 1, the vocabulary. Read api/types.go and api/client.go. This is the request and response grammar the whole system shares. Questions. What is on a ChatRequest, how does Options carry the sampler and context knobs, and how does the client turn a streamed body into a series of callbacks.

Stage 2, the front door. Read server/routes.go top to bottom, following one handler such as ChatHandler all the way through. Questions. Where is the model name resolved to a manifest, how are the baked-in template and parameters merged with the request, and where does the handler hand off to the scheduler.

Stage 3, packaging and distribution. parser for the Modelfile grammar, server/images.go for how commands become layers and a manifest, and server/download.go for the registry pull. Questions. What media type does each Modelfile instruction become, why is a blob named by its own hash, and why is the manifest written last during a pull.

Stage 4, the runtime core. server/sched.go, then the memory estimator around llm/memory.go and the GGUF reader that feeds it, then the discover package for GPU detection. Questions. How does a request become a resident runner, how is the GPU-layer count computed, which env vars bound concurrency and residency, and exactly when does keep-alive unload a model.

Stage 5, the backend boundary. The llm server launcher that spawns the subprocess, the llama cgo bindings, and the runner packages. Read this for the boundary, the private HTTP API between server and runner, not for stable file names, since this is the part mid-migration between the llama.cpp path and the native ml and model engine. Questions. What is on the runner's command line, what does /health gate, and where is the decode loop.

Stage 6, the edges. The openai compatibility package, the template package for prompt rendering, the convert package that imports Hugging Face safetensors into GGUF, and envconfig for every tunable. Questions. How is an OpenAI request translated, how does a Go template render chat messages into a prompt, and what does ollama create do when FROM points at a safetensors directory rather than an existing model.

Where not to start. The vendored llama.cpp and GGML C and C++ sources under llama and ml/backend are a whole world of their own, best read through the llama.cpp chapter rather than here. The native model engine is genuinely interesting but unstable and incomplete relative to the llama.cpp path, so meet it after the packaging and scheduling stories are solid.

Part VII: Hands-on labs

All labs need only the Ollama daemon and a small model such as llama3.2 or llama3.2:1b, which run on a laptop with or without a GPU. Log and field details shift with the fast pace of the project.

Lab 1: watch a model load and unload. Concept: the scheduler and keep-alive from Part IV and Part V.

ollama ps                                   # nothing loaded yet
ollama run llama3.2 "hello" > /dev/null      # loads on first use
ollama ps                                   # now resident, note the UNTIL column
# wait 5 minutes, or force it:
ollama stop llama3.2                         # unload now
ollama ps                                   # empty again

Then set OLLAMA_KEEP_ALIVE=-1 ollama serve in one shell and repeat. The model never expires on its own. Set keep_alive: 0 on a single request and watch it unload the instant that request finishes. You have just driven the refcount-plus-timer machine by hand.

Lab 2: see the content-addressable store. Concept: models are manifests over blobs, from the packaging deep dive.

ollama pull llama3.2
ls  ~/.ollama/models/manifests/registry.ollama.ai/library/llama3.2/
cat ~/.ollama/models/manifests/registry.ollama.ai/library/llama3.2/latest | python3 -m json.tool
ls -lh ~/.ollama/models/blobs/                # the big one is the GGUF image.model layer

Read the manifest and match each layer's mediaType to the Modelfile instruction it came from. Then run ollama cp llama3.2 twin and confirm the blob store did not grow, because the copy is a new manifest pointing at the same blobs.

Lab 3: build a custom model with a Modelfile. Concept: create compiles a Modelfile into layers.

# Modelfile
FROM llama3.2
PARAMETER temperature 0
SYSTEM "You are a JSON API. Reply only with a JSON object."

ollama create jsonbot -f Modelfile
ollama show jsonbot --modelfile             # see the resolved recipe
ollama run  jsonbot "give me a person with name and age"

Confirm with ollama show that your SYSTEM and PARAMETER layers are present, and that the FROM layer is the same image.model blob as the base, not a copy.

Lab 4: the OpenAI drop-in. Concept: /v1 is a shim over the native handlers.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
stream = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "count to five"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Run it, then run the equivalent against /api/chat with curl and compare the two wire formats for the same underlying generation. The tokens are produced by one handler and dressed two ways.

Lab 5: force the CPU cliff, then climb back. Concept: memory estimation and GPU offload from Part III and Part V.

# push the context large so the KV cache crowds the weights off the GPU
OLLAMA_DEBUG=1 ollama run llama3.2 --verbose "summarize the idea of entropy" \
  && echo "watch the debug log for the offload decision and layers on GPU vs CPU"

With OLLAMA_DEBUG=1 the daemon logs its memory estimate and how many layers it placed on the GPU. Try a larger num_ctx or a bigger model until layers spill to CPU and tokens per second collapse, then drop to a smaller quant tag or a smaller context and watch throughput recover. This is the single most useful intuition for running local models well.

Lab 6: embeddings for a tiny search index. Concept: the embed endpoint as a retrieval backend.

import ollama, numpy as np

docs = ["cats are mammals", "python is a language", "the sun is a star"]
E = ollama.embed(model="nomic-embed-text", input=docs)["embeddings"]
q = ollama.embed(model="nomic-embed-text", input="which animal?")["embeddings"][0]

sims = np.array(E) @ np.array(q)
print(docs[int(np.argmax(sims))])   # -> cats are mammals

Confirm the batch call returns one vector per document, then check that the embedding model stays resident under its own keep-alive in ollama ps, independent of any chat model.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is Ollama, in one sentence?

A small Go server that makes running an LLM locally a single command, by packaging models like container images, pulling them from a content-addressable registry, and supervising a llama.cpp/GGML runner subprocess through a memory-aware scheduler behind a native and OpenAI-compatible REST API.

2. Where does inference actually run, and why there?

In a separate runner subprocess, not in the daemon. The server spawns an ollama runner child that loads the GGUF and serves a private HTTP completion API, and the server proxies to it. The isolation means native code crashes take down a child rather than the daemon, weights never live in the server heap, and multiple models are just multiple processes that can be killed to free memory.

3. What are the two layers of the model store?

Manifests and blobs. A manifest is a Docker v2 JSON listing layer digests and media types for one model tag, and each blob is content stored under its own SHA-256 digest. A Modelfile compiles into those layers, and models that share a base share the base blob on disk.

4. Trace what happens on a pull.

Resolve the name, GET the manifest by tag, and for each referenced blob not already present GET it by digest over ranged, resumable, parallel chunks, verify each blob against its digest before committing it, then write the manifest last so a partial pull never looks installed. It is the standard OCI registry protocol.

5. What is a Modelfile and what does each instruction become?

A Dockerfile-shaped build recipe. FROM becomes the model weights layer, TEMPLATE the prompt template layer, SYSTEM the system-prompt layer, PARAMETER the default-options layer, ADAPTER a LoRA layer, and LICENSE a license layer, each a content-addressed blob under its own media type.

6. How does the scheduler decide whether it can load a model?

It asks the discover package for free VRAM per device, reads the GGUF metadata for per-layer sizes and layer count, sizes the KV cache from the requested context length and parallelism, and computes how many layers fit on the GPUs with room for the compute graph and overhead. The rest run on CPU, and if there is not enough room it evicts the least recently used idle runner within the OLLAMA_MAX_LOADED_MODELS limit.

7. Explain keep-alive precisely.

Each runner has a refcount and an expiry timer. A request increments the count and finishing decrements it, and when it reaches zero the timer arms for the keep-alive duration, five minutes by default. A duration keeps the model resident that long after the last request, 0 unloads immediately when the request finishes, and a negative value keeps it resident forever. This is why the second call is instant and an idle model still holds VRAM.

8. Why can a local model be slow with no error at all?

Because it partially offloaded. When the weights plus KV cache do not fit in VRAM, some layers run on the CPU at far lower memory bandwidth, so every token waits on those layers and throughput collapses while the model still produces correct output. The levers are a smaller quantization, a smaller num_ctx, or a smaller model, all of which free VRAM for more GPU layers.

9. How does the OpenAI-compatible API relate to the native one?

The /v1 routes are a translation shim in the openai package over the same native handlers. Middleware rewrites an OpenAI request into an api.ChatRequest or EmbedRequest, calls the identical handler the /api route uses, and rewrites the streamed response back into OpenAI's server-sent-events shape, so the compatibility is real rather than a separate engine.

10. What is the difference between /api/embed and /api/embeddings?

/api/embeddings is the legacy endpoint that takes a single prompt and returns one embedding. /api/embed is the newer one that takes an input string or array and returns a batch of embeddings. Prefer the batch form for indexing a corpus, and use a dedicated embedding model rather than a chat model.

11. When would you reach for llama.cpp, vLLM, or SGLang instead?

llama.cpp directly when you want every backend knob and the newest features first and are willing to manage GGUF files and flags. vLLM or SGLang when you are serving many concurrent users on a GPU and need PagedAttention and continuous batching for aggregate throughput. Ollama wins on zero-ceremony local single-user use, packaging, and a clean API, not on saturating a datacenter GPU.

12. What does the number after the colon in a model tag control, and why pin it?

The tag selects a specific variant, usually a size and quantization such as 8b-instruct-q4_0, and latest is whatever the registry currently points that tag at. A tag is mutable while a digest is immutable, so pin an explicit tag or digest when you need reproducible behavior rather than the moving default.

13. Why is the same binary both the CLI and the server and the runner?

A single static binary is the whole distribution story, with no runtime, environment, or dependency to install. ollama serve runs the daemon, the other verbs are HTTP clients to it, and a hidden ollama runner subcommand is what the daemon re-invokes as the isolated inference subprocess, so one artifact plays every role.

14. Your app's first request after an idle period is slow, then fast. Why?

The model was unloaded by keep-alive during the idle period, so the first request pays to reload the weights and rebuild the KV cache through the scheduler and a fresh runner subprocess, and subsequent requests hit the now-resident model. Raise keep_alive or set OLLAMA_KEEP_ALIVE to keep it warm, at the cost of holding VRAM.

Part IX: Design lessons

Package the artifact, not just the runtime. By compiling a Modelfile into content-addressed layers under a Docker manifest, Ollama got deduplication, integrity checking, resumable distribution, and a push-pull workflow without inventing any of it. The lesson is to borrow a proven distribution format wholesale, the same instinct behind language package registries and OCI images, rather than shipping a bespoke blob and a downloader.

Put the dangerous work in a child process. Native inference can crash or exhaust device memory, so Ollama runs it in a supervised subprocess behind a private HTTP API and keeps the daemon clean. Wherever code is fast but fragile, whether a codec, a parser, or a GPU kernel, isolating it behind a process boundary buys crash containment and clean teardown, the same pattern as a browser's per-tab processes.

Make the runtime memory-aware, not the user. The hardest part of local inference is fitting a model in the memory you have, and Ollama moves that decision into the scheduler by reading the GGUF, measuring free VRAM, and computing the offload split. Systems that own their resource math, rather than exposing a dozen flags and hoping, are the ones ordinary users can run.

Manage residency with a refcount and a timer. Keep-alive is nothing more than reference counting plus an expiry timer plus an eviction policy, and from those three pieces comes the entire feel of a responsive local model that also yields memory when idle. Connection pools, file caches, and JIT code caches all win with the same small state machine.

Meet the ecosystem where it already is. Shipping an OpenAI-compatible surface as a thin shim over the native handlers meant every tool built for that API worked against a local model on day one. A compatibility adapter over your real interface, done honestly, is often worth more than a superior but unfamiliar API of your own.

Part X: Memorization framework

The one-sentence summary: Ollama compiles a Modelfile into content-addressed layers, pulls them over the Docker registry protocol, and on demand a memory-aware scheduler spawns a llama.cpp/GGML runner subprocess, offloads as many layers as fit in VRAM, streams tokens back through a native and OpenAI-compatible API, and keeps the model warm until keep-alive expires.

cmd CLI -> api client -> server/routes (gin) -> handler
  -> Scheduler (sched.go): loaded? estimate VRAM, offload, evict, keep-alive
  -> LlamaServer (llm/server.go): spawn "ollama runner" subprocess
  -> runner: GGUF via cgo llama.cpp/GGML -> tokenize, decode, sample, detokenize
  -> stream tokens back up -> done, arm keep-alive timer

The chain mapped to source:

cli            cmd/cmd.go (cobra)
vocabulary     api/types.go, api/client.go
front door     server/routes.go (gin), openai/ (the /v1 shim)
packaging      parser/, server/images.go (layers + manifest)
distribution   server/download.go (registry pull), server/upload.go (push)
scheduling     server/sched.go, discover/ (GPU), llm/memory.go (offload)
runner         llm/server.go (spawn), llama/ (cgo), runner + ml/ + model/ (engine)

Memorize these blocks:

• Two big ideas: models are packaged like Docker images, and inference runs in a supervised subprocess.
• Store shape: a manifest of media-typed layers plus content-addressed blobs under sha256-<hex>, deduplicated across models that share a base.
• Pull: OCI registry v2, GET manifest then GET each missing blob, ranged and resumable, manifest written last.
• Scheduler: free VRAM from discover, layer sizes from GGUF, KV from num_ctx and parallelism, offload the rest to CPU, evict LRU idle within OLLAMA_MAX_LOADED_MODELS.
• Keep-alive: refcount plus expiry timer, default 5m, 0 unloads now, negative stays forever.
• API: native /api/generate, /api/chat, /api/embed, and the OpenAI shim /v1/* over the same handlers, on port 11434.

Part XI: Papers and further reading

The ideas this chapter leans on come from a handful of papers and specifications, and each one rewards a direct read. Where this site covers the same idea in depth, the companion link points there.

1. Vaswani et al., Attention Is All You Need, 2017. The transformer architecture every model Ollama runs is built on, and the origin of the KV cache the scheduler must budget for. The attention note on this site derives the mechanism.
2. Touvron et al., LLaMA, Open and Efficient Foundation Language Models, 2023. The open-weights release that sparked llama.cpp and with it the local-inference ecosystem Ollama packages. The language models from scratch class builds this kind of model end to end.
3. Grattafiori et al., The Llama 3 Herd of Models, 2024. The model family behind the llama3.1 and llama3.2 tags this chapter uses as its running examples.
4. ggml contributors, GGUF file format specification, 2023. The self-describing binary format of every image.model blob, metadata plus quantized tensors in one file. Covered in depth in the llama.cpp walkthrough.
5. Open Container Initiative, OCI Distribution Specification, 2021. The registry pull and push protocol Ollama speaks, standardized from the Docker Registry HTTP API V2, and the reason model distribution behaves exactly like container images.
6. Ollama maintainers, Modelfile reference. The authoritative grammar for FROM, PARAMETER, SYSTEM, TEMPLATE, ADAPTER, and LICENSE that the packaging deep dive decompiles into layers.
7. Dettmers and Zettlemoyer, The case for 4-bit precision, k-bit Inference Scaling Laws, 2022. The measurement showing 4-bit weights are close to optimal accuracy per byte, which is why the registry's default tags are 4-bit quantizations.
8. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The adapter format behind the ADAPTER instruction and its image.adapter layer. The PEFT walkthrough covers the method in practice.
9. Holtzman et al., The Curious Case of Neural Text Degeneration, 2019. Introduces nucleus sampling, the top_p knob carried in every options object alongside temperature and top_k.
10. Nussbaum et al., Nomic Embed, Training a Reproducible Long Context Text Embedder, 2024. The open embedding model behind nomic-embed-text in the labs. The sentence-transformers walkthrough covers embedding models more broadly.
11. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The high-throughput serving regime Ollama deliberately is not, covered in the vLLM walkthrough.

Part XII: Final takeaway

If the inference engine underneath is the gap, the llama.cpp chapter explains GGUF, block-wise quantization, and partial GPU offload that Ollama inherits, and the vLLM and SGLang chapters show the high-throughput serving regime Ollama deliberately is not. The broader design of a serving stack is drawn out in the LLM serving platform write-up, and the parallelism intuitions that scale this to a cluster live in the parallel computing notes. Then come back and read server/sched.go once more. The whole product is in that file.