Walkthroughs of the open source repositories worth knowing well: what each project does, how to use it, and how to actually read the code, starting from the entry points and following the main path through the modules that matter. Reading a great codebase is the fastest way to absorb its ideas, and each walkthrough is written to be the map I wish I had on a first visit.
Featured walkthroughHow to use vLLM in five minutes, the two ideas that make it fast, PagedAttention and continuous batching, and a guided path through the repository from the entry points to the scheduler and the CUDA kernels.
Read the walkthrough →The engines that turn a checkpoint into tokens per second. These walkthroughs pair with the LLM serving platform design write-up.
The reference open source LLM inference engine. Its two load-bearing ideas are PagedAttention, which manages the KV cache in fixed-size blocks the way an operating system pages virtual memory so almost no cache memory is wasted, and continuous batching, which admits and retires sequences every iteration instead of waiting for a batch to finish. The walkthrough covers usage, the engine loop, the scheduler, and where the kernels live.
The C/C++ engine that made local LLM inference normal. The walkthrough covers running a model in five minutes with llama-cli and llama-server, how the ggml tensor library, the GGUF file format, and block-wise quantization trade bits for bandwidth across CPUs and GPUs, and a guided reading path from the CLI tools down to the backend kernels.
The LMSYS serving framework built on the idea that the runtime and the programming interface should be co-designed. The walkthrough covers launching the OpenAI-compatible server, how RadixAttention turns prefix caching into the runtime's organizing principle, how jump-forward decoding makes structured output a fast path, what the frontend DSL adds, and a reading path through the SRT runtime.
NVIDIA's library for peak LLM inference on its own GPUs. The walkthrough covers the LLM API and trtllm-serve quickstart, the ahead-of-time engine-building philosophy and why the 1.0 release pulled it back into a PyTorch-native backend, in-flight batching in the C++ runtime, quantization formats co-designed with Hopper and Blackwell, and a reading path across the Python and C++ layers.
The tool that made running LLMs locally a single command, a small Go server wrapping a llama.cpp and GGML runner. Its two load-bearing ideas are packaging models like Docker images, where a Modelfile compiles into content-addressed layers pulled over the OCI registry protocol, and running inference in a supervised subprocess that a memory-aware scheduler loads, GPU-offloads, and keeps warm with a keep-alive timer. The walkthrough gets from install to a Python client in minutes, traces one ollama run request from the cmd CLI through server/routes, the scheduler, and the llm runner into the backend and back as streamed tokens, and covers the Modelfile format, pull mechanics, the native and OpenAI-compatible REST API, embeddings, and a guided reading path from the entry points.
The frameworks models are built and trained in, and the repos that defined how transformer code is written.
The framework nearly all of modern ML runs on, read from the training loop everyone knows down to the layers underneath it. The walkthrough follows what actually happens on loss.backward(), explains the dispatcher that routes every operator call across devices and features, sketches how torch.compile captures graphs without breaking eager code, and lays out a reading path through one of the largest repositories in open source.
Hugging Face's library of pretrained models, the standard interface between published checkpoints and the code that runs them. The walkthrough covers the Auto classes that resolve a Hub name to an architecture, the deliberately repetitive one-model-one-file philosophy that lets a single repo hold hundreds of readable architectures, and the pipeline and Trainer layers above, ending with a guided path through the source.
Tri Dao's FlashAttention kernels, the exact attention implementation that nearly every serious training and inference stack links or reimplements. The walkthrough explains why attention is bound by memory traffic rather than FLOPs, how tiling with online softmax lets the kernel stream over a score matrix it never materializes, and how the CUDA and CUTLASS code is organized across GPU generations. The math behind it is derived on the softmax page.
Andrej Karpathy's minimal GPT training repo: roughly three hundred lines of model and three hundred lines of training loop that reproduce GPT-2 (124M) on OpenWebText. The walkthrough trains a Shakespeare model in five minutes, reads model.py and train.py as a compact curriculum in real pretraining practice, and argues why this is the best first transformer codebase to read.
The PyTorch team's native platform for large-scale LLM pretraining, which trains Llama 3.1 from 8B to 405B by composing FSDP2, tensor, pipeline, and context parallelism over DTensor. The walkthrough launches a run in five minutes, explains how device meshes and per-parameter sharding make the parallelisms composable, and gives a reading path from the trainer loop through the distributed directory.
lm-evaluation-harness is EleutherAI's de facto standard for benchmarking language models and the engine behind the Hugging Face Open LLM Leaderboard. Its load-bearing idea is a seam between benchmarks and models, where every task reduces to three request primitives (loglikelihood, loglikelihood_rolling, generate_until) that any backend implements, so one YAML-defined benchmark runs identically against a Hugging Face model, vLLM, or a hosted API. Tasks are data rather than code, a config that names a dataset, a prompt template, answer choices, and metrics, with a version stamped into every result. The chapter traces one multiple-choice question from its YAML down to a single loglikelihood forward pass and back up through filters, metrics, and bootstrap aggregation, and it takes seriously why reproducible evaluation is genuinely hard, since prompt formatting, length normalization, answer extraction, chat templates, and contamination each move the number.
The systems that spread one model across many GPUs and many machines. These pair with the language model from scratch and parallel computing coursework.
Microsoft's training library that made trillion-parameter models feasible on hardware that could never hold one whole. Its load-bearing idea is ZeRO, the Zero Redundancy Optimizer, which notices that plain data parallelism keeps a full copy of the optimizer state, gradients, and parameters on every GPU and then partitions each across the group so per-GPU memory falls toward a single Nth of the total. Stage 1 shards the optimizer state, stage 2 adds the gradients, stage 3 adds the parameters, and ZeRO-Offload and ZeRO-Infinity spill those partitions down to CPU DRAM and NVMe. The walkthrough derives the ZeRO memory arithmetic from first principles, traces deepspeed.initialize and one ZeRO-3 step from the config through the just-in-time all-gather hooks to the CPU-side Adam step, and covers pipeline parallelism, the fused CUDA kernels and op_builder, and DeepSpeed-Inference, ending with a guided reading path and labs.
NVIDIA's reference implementation for training transformer language models at the largest scales, and the codebase where tensor, sequence, and interleaved pipeline parallelism were first published and shipped. The walkthrough explains why each transformer block is split column-parallel then row-parallel so a single all-reduce collects the result, how the conjugate f and g operators place exactly one collective per direction, how sequence parallelism shards the between-layer regions to save memory at no extra bandwidth, and how the 1F1B and interleaved pipeline schedules trade communication for a smaller bubble. It works through the communication-volume arithmetic for every parallelism dimension, the rule that keeps tensor parallelism on NVLink while pipeline and data parallelism cross nodes, and Megatron-Core as the reusable library under NeMo, ending with a reading path from pretrain_gpt.py through the tensor-parallel collectives to the gradient all-reduce, plus hands-on labs and understanding checks.
Hugging Face's thin layer that runs one PyTorch training loop unchanged on CPU, one GPU, many GPUs, or TPU. Its load-bearing idea is that the distribution strategy belongs to the launch environment rather than the code, so you add one Accelerator object, pass your model, optimizer, dataloader, and scheduler through a single polymorphic prepare() that adapts each to the current backend, and swap loss.backward() for accelerator.backward(loss). The walkthrough covers device placement, gradient accumulation, mixed precision, how prepare() wraps DDP, FSDP, and DeepSpeed behind one seam, the accelerate launch config and notebook_launcher, and a reading path from the Accelerator facade through the state singletons to the wrapped optimizer and dataloader.
Ray is the distributed compute substrate under much of modern ML infrastructure, turning ordinary Python into a cluster program through two primitives, stateless tasks and stateful actors, that communicate via an immutable shared-memory object store of futures. Its load-bearing ideas are ownership-based reference counting, where the worker that creates a reference owns its bookkeeping so metadata scales without a central master, and a single-controller model in which one driver orchestrates heterogeneous work rather than running SPMD lockstep. On that core sit the libraries teams actually use, Ray Data streaming blocks across CPU and GPU stages, Ray Train orchestrating a PyTorch DDP or FSDP job as fault-tolerant actors, Ray Serve composing model deployments behind autoscaling replicas, and RLlib running parallel samplers against a learner. Distributed scheduling flows through per-node raylets that grant worker leases and spill overflow, with a GCS control plane and placement groups for gang-scheduled GPU bundles. That single-controller shape plus placement groups is why Ray became the control plane for LLM RLHF, where verl and OpenRLHF colocate training and vLLM or SGLang generation as actors on one driver.
The PyTorch team's library for fine-tuning LLMs as readable recipes rather than a framework, where each run is a self-contained training script wired together by a small YAML config that names components by import path under a _component_ key. Its load-bearing ideas are the recipe/config split that keeps the training loop in plain sight and forkable, LoRA and QLoRA modules that freeze the base model (optionally as a 4-bit NF4 tensor from torchao) and train only tiny adapters, FSDP2 distributed recipes that differ from the single-device ones in just the sharding region, and a message-and-mask data path where prompt tokens are masked to -100 so the model learns only responses. The walkthrough runs real LoRA and QLoRA fine-tunes with the tune CLI, traces one lora_finetune_single_device run from tune run down to a checkpoint written back in the same Hugging Face format it was read, dives into the recipe and config design, the PEFT modules, and the dataset and prompt-template plumbing, and contrasts it honestly with axolotl's config-driven breadth over the HF stack and unsloth's single-GPU kernel speed.
Turning a pretrained checkpoint into an instruction-following, aligned, or reasoning model. These pair with the deep reinforcement learning and applied generative AI coursework.
Hugging Face's transformer reinforcement learning library, the primitive layer that post-training front ends like Axolotl, LLaMA-Factory, and Unsloth wrap. Its load-bearing idea is that each RLHF and preference-tuning algorithm is a thin subclass of the transformers Trainer that replaces only three things, a config extending TrainingArguments, a dataset preparation step, and a compute_loss, and inherits the entire training loop, gradient accumulation, mixed precision, checkpointing, and all. The walkthrough covers SFTTrainer's masking and packing, the DPOTrainer and its closed-form reward, the scaled log-ratio that lets a policy be its own reward so no reward model is needed, GRPOTrainer's critic-free group-normalized advantage, PPO with its value head, and Bradley-Terry reward modeling. It traces one DPO step end to end, shows how offline methods swap only the loss while online methods also generate and score, and ends with a guided read through the repository, labs, and understanding checks.
ByteDance's HybridFlow, the reinforcement-learning engine behind many open reasoning models. Its load-bearing idea is a single-controller program that orchestrates separate actor, rollout, critic, and reference worker groups, so one PPO or GRPO loop can pair an FSDP or Megatron training backend with a fast vLLM or SGLang rollout backend and place them on the same GPUs. The walkthrough covers the worker-group design, the Ray orchestration, and why co-locating generation and training is the throughput trick.
OpenRLHF is a clean, high-performance RLHF framework that treats reinforcement learning from human feedback as two cooperating distributed systems, running rollouts on dedicated vLLM engines and gradient updates on DeepSpeed ZeRO-3 actors while Ray places the policy, critic, reference, and reward models across the cluster. Its load-bearing idea is that generation dominates the wall-clock time of an RLHF step, so the highest-leverage move is to decouple it onto an inference engine and then reconnect the two systems by broadcasting the freshly trained policy weights back into the engines after every update. The same launcher runs PPO with a learned critic, GRPO and REINFORCE++ and RLOO that trade the critic for a statistical baseline, and reward signals that come from a trained model, an HTTP service, or an ordinary Python function, which is why verifiable-reward reasoning runs and classic preference RLHF share one harness. Colocation with vLLM and DeepSpeed sleep modes fits every role onto a single node, while disaggregated placement plus async rollouts reclaim overlap at cluster scale.
Hugging Face's parameter-efficient fine-tuning library, the standard way to adapt a large model by training a tiny fraction of it. Its load-bearing idea is a config-driven wrapper, get_peft_model, that injects small trainable modules into a frozen base, with LoRA's low-rank update as the default and QLoRA adding a 4-bit base so a single GPU can fine-tune a large model. The walkthrough derives the LoRA parameter savings, traces how the adapter attaches to a linear layer, and covers merging adapters back for inference.
Unsloth makes LoRA and QLoRA fine-tuning roughly twice as fast in a fraction of the VRAM without changing the math, and it does it by rewriting only the hot paths. A transformer fine-tuning step spends nearly all of its time and memory in a short list of operations, so Unsloth replaces rotary embeddings, RMSNorm, the gated SwiGLU MLP, and the cross-entropy loss with hand-derived torch.autograd.Function classes backed by Triton kernels, each fusing the whole operation into one launch and writing its backward by hand so it saves the minimum (a reciprocal std, the cos/sin tables, one log-sum-exp per token) and recomputes the rest. The base weights load in 4-bit and stay frozen while tiny LoRA adapters train, the dequantize-matmul-adapter chain is fused so transient full-precision weights never become resident, and an offloaded gradient checkpointer streams block activations to pinned CPU RAM to decouple peak memory from sequence length. Rather than fork the model it monkey-patches the live Hugging Face transformers classes at import time, so PEFT, TRL, generate, and GGUF export all keep working, and because the kernels are exact rather than approximate the accuracy is unchanged. It is a single-GPU accelerator built to make a real fine-tune fit on the smallest card that could plausibly work.
The config-first fine-tuning framework from Axolotl AI, where a single validated YAML file drives supervised fine-tuning, preference tuning like DPO and KTO, GRPO reinforcement learning, and reward modeling across dozens of model families, all compiled down to Hugging Face transformers, PEFT, TRL, and DeepSpeed. Its load-bearing idea is that the YAML surface is the product, a Pydantic schema turns fine-tuning expertise into rules the machine enforces and a run into a diffable document, while prompt strategies handle formatting and loss masking, sample packing with boundary-aware flash attention removes padding waste, and sequence parallelism plus ZeRO or FSDP scale runs out from outside the model. The walkthrough covers the quickstart, dataset formats and prompt strategies, the full life of one LoRA fine-tune from CLI to saved adapter, the internals of packing, the trainer builder, and sequence parallelism, and a guided reading path through the source.
A unified fine-tuning hub that turns adapting a base model into a single config file, reachable from a CLI, a Python call, or the LlamaBoard web UI. Its load-bearing ideas are two registries, a template registry that captures each model family's chat format and fixes the tokenizer, and a JSON dataset registry whose aligner normalizes alpaca and sharegpt data into one internal schema, together with a single run_exp dispatch that switches on the stage into six workflows covering SFT, reward modeling, PPO, DPO, and KTO, each delegating the actual loop to a lightly subclassed Hugging Face or TRL trainer. The walkthrough covers a real quickstart, the full life of one LoRA SFT run from llamafactory-cli through the trainer dispatch to a saved adapter, deep dives on the template and dataset registries and the model and adapter layer, a guided reading path, and labs. Because it orchestrates Transformers, PEFT, and TRL rather than reimplementing them, it lowers the barrier to a first fine-tune across more than a hundred models while leaving pretraining at scale to torchtitan and production serving to vLLM.
Where the FLOPs actually come from, and the sequence architectures reaching past attention.
Triton, the Python DSL and MLIR-based compiler that made high-performance GPU kernels writable without CUDA C++, and the code-generation backend behind PyTorch's torch.compile. The walkthrough builds a vector-add and a fused softmax kernel from nothing, explains the block/tile programming model where you index whole tiles with program_id and let the compiler own thread mapping, coalescing, shared memory, and synchronization, and traces one kernel launch from @triton.jit through Triton IR, Triton GPU IR (where layouts are chosen), LLVM, and PTX down to a launched grid. It covers autotuning, block pointers, and tl.dot, and gives a reading path from the official tutorials through the runtime's JIT and specialization cache to the MLIR passes and vendor backends.
CUTLASS is NVIDIA's open library of CUDA C++ templates for assembling matrix-multiply and convolution kernels that run at close to peak tensor-core throughput. Its load-bearing idea is a hierarchical decomposition of one GEMM into threadblock, warp, and instruction tiles, one per level of the GPU memory system, so that each level reuses data enough times to keep the tensor cores fed, an argument that falls straight out of the roofline and arithmetic-intensity math. Underneath sits CuTe, a small algebra of shapes and strides where a Layout is just a function from coordinate to offset, which turns the thread-to-data mapping into a value you can compose and lets MMA and copy atoms, the pipelined mainloop, and the fused epilogue all be written as tiled layouts. The chapter traces one tensor-core GEMM from the host launch through the cp.async or TMA loads and the wgmma math up to the Epilogue Visitor Tree, and shows how FlashAttention and inference engines like vLLM and TensorRT-LLM build their quantized and attention kernels on the same atoms.
Albert Gu and Tri Dao's reference implementation of the selective state-space model, the architecture that runs sequence models in linear time and constant inference memory while staying competitive with Transformers. The walkthrough builds the one load-bearing idea, selectivity, making the state-space parameters B, C, and the step size Delta functions of the input so a fixed-size hidden state can choose what to remember, then shows why that input-dependence forbids the convolutional view every earlier state-space model relied on and forces a hardware-aware CUDA scan kernel that keeps the oversized state on chip and recomputes it in the backward pass. It traces one forward pass through the Mamba block from in_proj through the causal convolution, the S6 selective scan, and the SiLU gate, and covers Mamba-2, whose scalar state matrix exposes a duality with masked attention and reshapes the work into tensor-core matmuls. The theory behind the discretization, the associative scan, and the duality is derived on the state-space models class page.
Language models wired into loops that plan, use tools, and act, including agents that run their own research.
autoresearch is Andrej Karpathy's small, deliberately toy-sized experiment in agentic ML research, where a general coding agent runs its own overnight research loop on a single-file, single-GPU language-model training script. Its load-bearing idea is an inversion of who writes what: the human authors program.md, a markdown file of research-org policy that states the goal, the rules, the accept criterion, and a NEVER STOP directive, while a language model like Claude Code executes that policy by editing train.py for you, so you program the context and the agent programs the Python. The loop is a strict ratchet: hypothesize, edit the one editable file, train for exactly five minutes of wall clock, read one fixed metric (validation bits per byte, byte-normalized so tokenizer changes cannot game it), and keep the git commit only if the number improved, otherwise revert. A three-file contract keeps it honest, with prepare.py holding the data, tokenizer, budget, and metric as a read-only referee that the agent may never touch. The design is safe and legible precisely because it is greedy and tiny, which is also its ceiling: it is a diligent tireless tweaker that stacks small wins on already-tuned code, not an inventor of new architectures.
An autonomous agent that turns a single question into a long, cited research report. Its two load-bearing ideas are the planner and executor split, where a strategic reasoning model decomposes the question into a handful of sub-queries and many independent executors research them in parallel with asyncio.gather, and a two-method boundary that hides every search engine and every scraper behind pluggable factories chosen by config name. Along the way it deduplicates URLs against a run-wide visited set, compresses scraped pages down to the passages that matter by embedding similarity, and lets a separate writer model synthesize the pooled evidence into prose with inline sources. The walkthrough covers the quickstart across the CLI, the pip package, and the FastAPI server, the full life of one research task traced end to end, deep dives on the skills and actions layers, the three LLM roles and the config knobs that set cost and breadth, a staged reading path through the repository, and hands-on labs.
AutoGen is Microsoft's framework for building applications out of multiple conversing LLM agents, where a hard task is solved not by one completion but by a structured conversation among a planner, a coder, a tool executor, a critic, and a human. Its enduring abstraction is the conversable agent, an entity that sends messages, receives messages, and generates replies through a pipeline of handlers, so code execution, tool calls, and human approval are all ordinary turns rather than special cases. The v0.4 rewrite kept that programming model and rebuilt the machinery as an event-driven actor core, autogen-core, where agents are identities that exchange typed messages by direct send or topic broadcast, with the high-level AgentChat API of teams, group chat, and composable termination conditions layered on top and autogen-ext supplying model clients, code executors, and a cross-language distributed gRPC runtime. The load-bearing ideas are the message as a universal interface, the actor model for isolation and location transparency so the same agent code runs in one process or across a cluster, and feedback loops that ground language models in real results, which is why multi-agent conversation can do what a single agent cannot.
LangChain's framework for building agents as explicit stateful graphs rather than an implicit loop. Its load-bearing idea is a graph of nodes and conditional edges over a typed shared state, with a checkpointer that persists every step so a run can stream, pause for human input, and resume or time-travel. The walkthrough covers the graph and state model, the checkpointer, human-in-the-loop interrupts, and why a graph gives control a plain agent loop cannot.
Stanford NLP's framework for programming foundation models instead of prompting them, built on the idea that you should specify what a language-model step does and let the framework write the prompt. Its load-bearing move is to separate behavior, a typed signature such as question to answer wired together by modules like Predict, ChainOfThought, and ReAct, from the exact prompt text, which an optimizer then compiles by bootstrapping few-shot demonstrations from the program's own successful runs and proposing grounded instructions, all scored against a metric you define. The walkthrough configures a model and writes a first program in a few lines, then traces the full life of one Predict call from signature to chat messages to parsed prediction and one BootstrapFewShot compile from a training set to learned demonstrations. It covers the optimizer family from LabeledFewShot through MIPROv2 and BootstrapFinetune, the metric contract that doubles as a scoring function and a demo gate, and a staged reading path through the package from signatures to teleprompters, ending with labs, understanding checks, and design lessons.
Princeton NLP's language-model agent that autonomously fixes GitHub issues, and the reference implementation built by the same group behind the SWE-bench benchmark. Its load-bearing idea is the Agent-Computer Interface, the finding that a compact, stateful, guarded interface rather than the model behind it is what drives the pass rate, realized as a small command set of windowed file navigation, summarizing search, and a file editor that runs flake8 after every edit and refuses changes that introduce new errors. The walkthrough runs the agent on a real issue with sweagent run, traces one issue end to end from config resolution through the SWE-ReX sandbox and the thought-action loop to a git-diff patch scored by the SWE-bench harness, and gives deep dives on tool bundles, the state registry, the linting editor, and FAIL_TO_PASS versus PASS_TO_PASS grading, with a reading path, labs, and understanding checks.
The libraries behind image, video, and audio generation.
huggingface/diffusers is the standard library for diffusion models, the place where new samplers, backbones, and adapters usually land first and where Stable Diffusion, SDXL, DiT, and Flux all share one set of interfaces. Its load-bearing idea is a three-way separation. A model is a plain denoiser, a scheduler is the weightless sampling math, and a pipeline is the recipe that wires them to a text encoder and a VAE and runs the loop, so a backbone, a sampler, and a task recipe can each be swapped in isolation. A uniform config-plus-weights contract shared by ConfigMixin, ModelMixin, SchedulerMixin, and DiffusionPipeline lets any combination round-trip through the Hub, which is why from_config can rebuild a sibling scheduler for free and why the same code path serves images, video, and audio. The page walks the scheduler zoo from ancestral DDPM through DDIM, DPM-Solver, and flow matching, the UNet and DiT backbones, and adapters like ControlNet and LoRA that add control by composition rather than by forking, then traces one text-to-image call from from_pretrained through the denoising loop to the decoded image.
A node-graph engine for diffusion pipelines that exposes every stage of image generation, model loading, text conditioning, sampling, and VAE decoding, as a typed node you wire together by hand rather than hiding it behind a one-click form. Its two load-bearing ideas are nodes as typed pure functions over data types like MODEL, CONDITIONING, and LATENT, and an executor that runs the graph back to front from its output nodes while caching each node output on the recursive signature of its inputs, so editing one prompt recomputes only what changed downstream. The ModelPatcher applies LoRAs by cloning a wrapper that shares weights, so patches never mutate or invalidate the cached checkpoint, and the whole custom-node ecosystem exists because a node is just a class with four attributes and a method. The walkthrough covers running it and writing a node, the full life of one text-to-image generation from Queue Prompt through the executor, cache, sampler, and decode, deep dives on the execution graph and the ModelPatcher, why exposing sampling as dataflow gives power users control a one-click UI cannot, and a reading path from main.py through the comfy execution and diffusion core.
Giving a model the right context, through embeddings, vector retrieval, and the data frameworks around them.
LlamaIndex is a data framework for building LLM applications over your own documents, and its one load-bearing abstraction is the Node, a chunk of content carrying metadata, typed relationships, and an embedding, so that ingestion, indexing, retrieval, and answer synthesis all become operations over lists of nodes. Its second key idea is a clean split between retrieval and synthesis, where a query engine is precisely a retriever plus node postprocessors plus a response synthesizer, each swappable, which is why the same handful of pieces covers naive RAG, reranked RAG, multi-document routing, and SQL question answering. The chapter traces one call to query_engine.query() down through the retriever and the vector store and back up through response modes like compact, refine, and tree_summarize, covers the vector, summary, and keyword indices and the StorageContext behind them, and reaches the newer event-driven Workflow and agent layer where agents are just workflows that call query engines as tools. It maps the run-llama/llama_index monorepo from llama-index-core through the separately versioned integration packages, flags the v0.10 refactor and the Settings-over-ServiceContext migration, and ends with runnable labs and understanding checks.
The library that made text embeddings practical, grown out of the UKP Lab's 2019 Sentence-BERT work. Its load-bearing ideas are the bi-encoder, which reads each text once into a single fixed-size vector so comparison becomes a cheap dot product over precomputed vectors instead of a forward pass per pair, mean pooling that turns per-token embeddings into a sentence vector by an attention-masked average, and MultipleNegativesRankingLoss, which trains that vector contrastively using every other example in the batch as a free negative. The walkthrough encodes and searches in a few lines, traces one encode() call through the Transformer, Pooling, and Normalize module pipeline to a numpy array, explains the contrastive losses and the Hugging Face-based SentenceTransformerTrainer, and shows how cross-encoder rerankers pair with bi-encoder retrieval to form the retrieve-then-rerank engine under semantic search and RAG, ending with a reading path from SentenceTransformer.py through the losses to the cross-encoder.
The storage engines most software stands on, and some of the most readable systems code ever published.
The most widely deployed database in the world, developed in Fossil at sqlite.org with a read-only GitHub mirror. The walkthrough covers how a prepared statement becomes bytecode for the VDBE virtual machine, how the B-tree, pager, and WAL stack beneath it, and the amalgamation build and avionics-grade testing culture that let it run unattended on billions of devices, ending with a reading path from parse.y down to wal.c.
The in-memory data structure server, and the clearest demonstration that a single-threaded event loop is a performance strategy rather than a limitation. The walkthrough covers the object system's adaptive encodings, listpacks that grow into hash tables and skiplists, RDB and AOF persistence via fork and copy-on-write, and a reading path that follows one SET command from server.c through ae.c and networking.c to the reply.
The database whose old decisions still carry it: MVCC that never updates rows in place and pays for it with vacuum, a write-ahead log that unifies crash recovery, replication, and point-in-time restore, and a process-per-connection model with catalog-driven extensibility. The reading path walks the query pipeline itself, from postmaster.c through the parser, planner, and executor to the storage and WAL layers.
The LSM tree in production form, from Meta's database engineering team and built on LevelDB. The walkthrough covers embedding it in a C++ program, how memtables, the write-ahead log, and immutable SST files turn random writes into sequential I/O, why choosing a compaction style means choosing between write, read, and space amplification, and a reading path from the public headers through the write path and compaction pickers. It pairs with the key-value store design write-up.
How things are found and how columns move: the indexing and analytics engines behind retrieval systems like the ones in the systems section.
The Java search library under Elasticsearch, OpenSearch, and Solr, started by Doug Cutting in 1999. The walkthrough indexes and searches in five minutes with the core API, then explains the inverted index and the immutable-segment model that gives Lucene its lock-free reads, how postings and the codec layer encode the index on disk, and why BM25 has been the default scoring model since Lucene 6, with a reading path through IndexWriter and the search execution chain. It pairs with the search service design write-up.
Meta's C++ library for similarity search over dense vectors, the production home of the canonical ANN index structures. The walkthrough covers exact and approximate search in five minutes, the flat, IVF, HNSW, and PQ families as points on the recall-speed-memory triangle, the index factory string that composes them, how to choose an index for a real workload, and how to read the C++ core.
An in-process analytical database from the CWI research institute in Amsterdam, often summarized as SQLite for analytics. The walkthrough gets from install to querying a Parquet file in one command, then explains columnar-vectorized execution, the morsel-driven parallelism idea borrowed from the HyPer literature, and how replacement scans and pushdown let it query data where it lives, ending with a reading path through the parser-to-pipeline source tree.
The servers and frameworks between a request and an answer.
The Python framework where a typed function signature becomes validation, serialization, and live OpenAPI docs from a single declaration. The walkthrough covers the five-minute path from install to interactive docs, how the framework is really a thin layer of glue over Starlette's ASGI machinery and Pydantic's validation, how dependency injection works as plain function parameters, and a reading path through applications.py, routing.py, and the dependencies package.
Igor Sysoev's answer to the C10K problem and still the reverse proxy fronting much of the web. The walkthrough covers a minimal reverse-proxy setup, the master/worker event-loop architecture that makes reloads graceful and blocking forbidden, the eleven-phase request pipeline that lets modules compose without knowing about each other, and a reading path from main() through the process cycle to the phase engine.
Built around the one design pattern the entire codebase repeats: level-triggered reconcile loops converging observed state toward desired state stored behind the API server in etcd. The walkthrough covers a five-minute kind cluster where you delete a pod and watch the system win, how components coordinate only through shared state, why client-go informers make a thousand watchers cheap, and an honest reading path that skips the front door for sample-controller, the replica set controller, and the scheduler's fit plugins.