Applied generative AI, from adaptation and retrieval to agents and serving

Why this subject matters now

Five years ago the applied task was training a model. Today, for all but a handful of labs, the base model is a given, a frozen checkpoint downloaded from a hub, or a hosted endpoint behind an API. The value has moved to everything that surrounds it. A practitioner is now expected to adapt a base model to a domain on a single consumer GPU, ground its outputs in a private corpus so it answers questions about documents it never saw in pretraining, force its output to satisfy a JSON schema so downstream code can parse it, wire it into an agent loop that calls tools and reacts to their results, and then serve the whole thing at a cost-per-token that a business can absorb. Each of these has become its own body of technique with its own literature, and the techniques interact. Quantization changes what fits in the KV cache, which changes the batch size, which changes the cost model.

What changed to make this possible was a cluster of results, most from 2021 through 2023, that each attacked a different bottleneck. Low-rank adaptation (Hu et al., 2021) showed that the weight update during fine-tuning has low intrinsic rank, so it can be trained as a pair of thin matrices while the base stays frozen. QLoRA (Dettmers et al., 2023) pushed the frozen base to four-bit precision with a data type matched to the Gaussian shape of weights, bringing 65B fine-tuning onto one 48GB card. Direct preference optimization (Rafailov et al., 2023) collapsed the reinforcement-learning alignment pipeline into a single supervised loss. Retrieval-augmented generation (Lewis et al., 2020) separated parametric knowledge from a non-parametric store that can be edited without retraining. Constrained decoding (Willard and Louf, 2023) made grammar-conformant output a compile-time index rather than a sampling gamble. And paged attention (Kwon et al., 2023) made the memory of serving cheap enough that continuous batching became standard. This page treats each of these as a derivation, not a recipe. The full reinforcement-learning treatment of alignment, including PPO and GRPO, lives in the deep reinforcement learning page. The vector-index internals (HNSW, IVF) live in the databases page. Agent self-improvement loops live in self-improving agents. Here we cross-link those and derive what sits between them.

Core theory

The low-rank hypothesis and LoRA

Full fine-tuning updates every weight. For a linear layer with weight \( W_0 \in \R^{d \times k} \), gradient descent produces \( W = W_0 + \Delta W \) with \( \Delta W \) a dense \( d \times k \) matrix, and it stores an optimizer state for every one of those \( dk \) entries. The empirical observation behind LoRA is that the useful part of \( \Delta W \) lives in a low-dimensional subspace. The model was already competent after pretraining, and adapting it to a task nudges it along a handful of directions rather than reshaping every coordinate independently. Aghajanyan et al. (2020) measured this directly, finding that large pretrained models can be fine-tuned in a randomly projected subspace of only a few hundred to a few thousand dimensions and still reach most of full-fine-tuning quality. LoRA takes that as a structural prior on the update.

The reparametrization is to constrain the update to rank \( r \ll \min(d,k) \) by writing it as a product of two thin matrices,

The base \( W_0 \) is frozen. Only \( A \) and \( B \) receive gradients. The forward pass on input \( x \in \R^{k} \) is \( h = W_0 x + \frac{\alpha}{r} B (A x) \), computed as two small matrix-vector products added to the frozen path, so no dense \( d \times k \) matrix is ever materialized for the update. At initialization \( A \) is drawn from a small random Gaussian and \( B \) is set to zero, so \( \Delta W = 0 \) and the adapted model is identical to the base on the first step. Training then grows the update from nothing, which keeps the early optimization well-conditioned and means an untrained adapter never degrades the base. The scalar \( \alpha/r \) is a fixed scaling, not a learned parameter. Its purpose is to decouple the learning rate from the rank. If you double \( r \), the singular values of a randomly initialized \( BA \) grow roughly like \( \sqrt{r} \), and dividing by \( r \) keeps the update magnitude comparable across ranks so a tuned learning rate transfers. In practice people set \( \alpha \) to a constant (often \( \alpha = 2r \) or \( \alpha = r \)) and forget it.

Why does this save so much? The trainable parameter count drops from \( dk \) to \( r(d + k) \). For a square projection \( d = k = 4096 \) and \( r = 16 \), that is \( 16{,}777{,}216 \) parameters down to \( 16 \times 8192 = 131{,}072 \), a factor of 128. But the headline number in a fine-tuning budget is not the parameter count, it is the optimizer state. Mixed-precision AdamW keeps, per trainable parameter, a 16-bit weight and gradient plus three 32-bit tensors (a master copy of the weight, the first moment \( m \), and the second moment \( v \)), totalling \( 2 + 2 + 4 + 4 + 4 = 16 \) bytes (the accounting is the one Rajbhandari et al., 2019, use in the ZeRO paper). Full fine-tuning of a 7B model therefore needs \( 16 \times 7 \times 10^9 = 112 \) GB of optimizer and model state alone, before activations, which does not fit on an 80GB card. LoRA keeps the frozen base in 16-bit (\( 2 \times 7\text{B} = 14 \) GB, read-only, no optimizer state) and applies the 16-byte treatment only to the adapters. With adapters on the query and value projections of all 32 layers at \( r = 16 \), the adapter count is \( 2 \times 32 \times 16 \times (4096 + 4096) = 8{,}388{,}608 \) parameters, whose optimizer state is \( 16 \times 8.39 \times 10^6 \approx 134 \) MB. The trainable footprint went from 112 GB to 134 MB. The total (base plus adapters) is about 14.1 GB. That is the whole argument for why LoRA fits where full fine-tuning does not, and Problem 1 works it in detail.

Two properties follow that matter in production. First, the adapter is mergeable. At inference you can fold \( W_0 \gets W_0 + \frac{\alpha}{r}BA \) once and serve with zero added latency, because the merged model has the base's exact shape. Second, adapters are composable and swappable. One frozen base in memory can serve many tasks by hot- swapping small adapter tensors, which is the basis of multi-tenant LoRA serving (S-LoRA, Sheng et al., 2023). The cost is that a merged adapter is task-specific. Serving many un-merged adapters simultaneously reintroduces the small \( BA \) matmul per request.

QLoRA with NF4, double quantization, and paged optimizers

LoRA froze the base in 16-bit. QLoRA asks whether the frozen base can be stored in four bits without hurting the gradients that flow to the adapters, and answers yes, with three ideas.

NF4 (4-bit NormalFloat). A naive 4-bit integer quantizer places its 16 levels uniformly across \( [-\text{absmax}, +\text{absmax}] \). But neural-network weights are not uniform. Within a block they are approximately zero-mean Gaussian. Uniform levels waste resolution in the tails where few weights live and starve the center where most do. NF4 instead places its levels at the quantiles of a standard normal, so each of the 16 bins holds an equal probability mass of a Gaussian rather than an equal width. Concretely, the level values are chosen as \( q_i = \frac{1}{2}\big( Q_N(\tfrac{i}{17}) + Q_N(\tfrac{i+1}{17}) \big) \) style quantile midpoints of \( Q_N \), the standard-normal inverse CDF, then rescaled so the outermost levels sit at \( \pm 1 \). This makes NF4 information-theoretically optimal for exactly-Gaussian inputs (it is the Lloyd-Max quantizer for the normal density under the constraint of 16 fixed levels), and it is symmetric with an exact zero. To quantize a real weight block, divide by the block's absolute maximum so it lands in \( [-1, 1] \), then snap to the nearest NF4 level and store the 4-bit index plus one scale per block.

Double quantization. Blockwise quantization with block size 64 stores one 32-bit scale per 64 weights. That scale costs \( 32/64 = 0.5 \) bits per weight, so the true cost of NF4 is not 4 but 4.5 bits per weight. Double quantization quantizes the scales themselves. The fp32 block scales are grouped (block size 256) and stored as 8-bit values with their own single fp32 scale per group. The per-weight cost becomes

down from 4.5, a saving of about 0.37 bits per weight, or roughly 3 GB on a 65B model for free. The 8-bit re-quantization of scales is nearly lossless because scales vary slowly.

Paged optimizers. Even with a frozen 4-bit base, a long-sequence backward pass produces gradient-memory spikes that can momentarily exceed the card. QLoRA uses NVIDIA unified memory to page optimizer state between GPU and CPU RAM on demand, the same mechanism the OS uses for virtual memory, so a transient spike evicts cold pages instead of triggering an out-of-memory kill. This is a robustness trick, not a compression trick. It turns rare OOM crashes into graceful slowdowns.

The dequantization happens on the fly during the forward and backward passes. The 4-bit base weight is expanded to the compute dtype (bf16), used in the matmul, and discarded, so the base never occupies more than its 4-bit footprint in storage. Gradients flow through the frozen, dequantized weights to the LoRA adapters, which are kept in bf16 and are the only trainable parameters. The full memory accounting for a 7B model is worked in Problem 1. The QLoRA base is about 3.5 GB, versus 14 GB for LoRA's 16-bit base and 112 GB of state for full fine-tuning.

Adapters and prefix tuning, briefly

LoRA is one point in a family of parameter-efficient methods. The original adapter (Houlsby et al., 2019) inserts small bottleneck modules, a down-projection to dimension \( m \ll d \), a nonlinearity, an up-projection back to \( d \), plus a residual, after each sub-layer, \( h \gets h + f(h W_{\text{down}}) W_{\text{up}} \). It adds \( 2dm \) parameters per insertion and, unlike LoRA, cannot be merged into the base, so it costs a small fixed latency at inference. Prefix tuning (Li and Liang, 2021) and the related prompt tuning prepend a set of trainable "virtual token" vectors to the keys and values at every layer. The model attends to these learned prefixes as if they were context, and only the prefix vectors are trained. Prefix methods shine when you want many lightweight task specializations of one served base, because a prefix is just a small tensor of KV vectors, but they consume context budget and are generally less expressive per parameter than LoRA on hard adaptation tasks. The unifying view (He et al., 2021) is that all three inject a low-cost, learned perturbation into a frozen network at a chosen location. They differ in where they inject and whether the injection folds back into the base.

Instruction tuning and data curation

A base model trained only on next-token prediction over web text will continue a prompt rather than answer it. Asked "List three uses of a paperclip," it may produce a plausible-looking multiple-choice question instead, because that is a common web pattern. Instruction tuning is supervised fine-tuning on (instruction, response) pairs that teaches the model to treat the prompt as a request. FLAN (Wei et al., 2021) and its scaled successor (Chung et al., 2022) showed that fine-tuning on a large, diverse mixture of tasks phrased as instructions produces strong zero-shot generalization to held-out task types, and that the number of distinct tasks matters more than the number of examples per task. The mechanism is not new knowledge. It is format alignment, teaching the model the mapping from an instruction surface form to the answer-generating behavior it already latently has.

Because the objective is imitation, data quality dominates. The LIMA result (Zhou et al., 2023) is the sharp version of this claim. A strong base fine-tuned on only 1,000 carefully curated, diverse, high-quality examples matched models trained on orders of magnitude more, supporting a "superficial alignment hypothesis" that most capability is learned in pretraining and instruction tuning mainly selects a response style. The practical consequences are that deduplication (near-duplicate instructions inflate apparent dataset size without adding signal), diversity of task types, and removing low-quality or templated responses buy more than raw scale, and that a handful of mislabeled or sycophantic examples can teach a persistent bad behavior because the model is imitating, not averaging. Curation is where most instruction-tuning effort actually goes.

Alignment, from the RLHF pipeline to the DPO closed form

Instruction tuning gets format. It does not get preference. To make a model prefer helpful, harmless, honest responses when several completions are all fluent, the standard pipeline is reinforcement learning from human feedback (Ouyang et al., 2022, InstructGPT). It has three stages. First, supervised fine-tuning on demonstrations, producing a reference policy \( \pi_{\text{ref}} \). Second, a reward model. Collect human preference pairs \( (x, y_w, y_l) \) where \( y_w \) is preferred to \( y_l \), and fit a scalar reward \( r_\phi(x,y) \) under the Bradley-Terry likelihood \( \P(y_w \succ y_l \mid x) = \sigma\big( r_\phi(x,y_w) - r_\phi(x,y_l) \big) \). Third, optimize the policy to maximize expected reward while staying close to the reference, penalizing drift with a KL term so the policy does not collapse onto degenerate high-reward text,

The full treatment of stage three, PPO's clipped surrogate, generalized advantage estimation, and GRPO's value-free variant, is derived on the deep reinforcement learning page. That page owns the RL. Here we derive only the result that lets you skip stage three entirely.

DPO (Rafailov et al., 2023) observes that the KL-constrained objective has a closed-form optimum. Fixing a prompt \( x \), the inner maximization over the response distribution \( \pi \) is solved by a Boltzmann tilt of the reference,

This is standard, the variational solution to a KL-regularized reward maximization. The step-by-step proof, completing the square as a KL against \( \pi^* \), is on the deep-RL page. The DPO move is to run it backwards. Solving for the reward,

so every reward is a scaled log-ratio between its own optimal policy and the reference, plus a prompt-dependent constant \( \beta \log Z(x) \). Substituting into the Bradley-Terry likelihood, the intractable partition term cancels because it is the same for \( y_w \) and \( y_l \) at a shared \( x \). Reparameterizing the unknown optimal policy directly as the trainable \( \pi_\theta \) and doing maximum likelihood on the preference data gives the DPO loss, restated here for reference,

No reward model, no sampling from the policy, no RL loop. The policy is its own implicit reward model, trained by supervised classification on which completion humans preferred. Writing the implicit reward \( \hat r_\theta(x,y) = \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y \mid x)} \), the gradient is

which raises the chosen completion's log-probability and lowers the rejected one's, weighted by how wrong the current implicit reward is. The weight \( \sigma(\hat r_\theta(x,y_l) - \hat r_\theta(x,y_w)) \) is near 1 when the model has the pair backwards and decays toward 0 once the pair is ranked with margin. Problem 2 puts numbers on this. What DPO gives up relative to online RLHF, the on-policy state distribution and the ability to explore beyond the preference dataset, is discussed on the deep-RL page. The two are complementary, and many production pipelines use DPO as a cheaper replacement or a warm start for a shorter RL phase.

Quantization for serving

Training quantization (QLoRA) compresses a frozen base so it fits during fine-tuning. Serving quantization compresses a finished model so it is cheaper and faster to run, and the constraints are different. Latency and throughput matter, and the quantized model must stay accurate under real inputs. The unit of the decision is where the low precision goes.

Weight-only vs activation quantization. Decode is memory-bandwidth bound (derived in the serving section). Each generated token must read every weight from HBM. Halving the weight bytes nearly halves decode latency at small batch, independent of whether the arithmetic is faster. Weight-only quantization (store weights in 4-bit, dequantize to bf16 for the matmul) therefore buys latency even though the matmul itself runs in bf16. Activation quantization additionally stores and multiplies activations in low precision (INT8 or FP8), which unlocks faster tensor-core paths (INT8 and FP8 matmul run at roughly twice bf16 on H100), but activations have outliers that weights do not, and quantizing them badly destroys accuracy.

The outlier problem, and LLM.int8(). Dettmers et al. (2022) found that beyond about 6.7B parameters, transformer activations develop systematic outlier features, a few hidden dimensions with magnitudes 10 to 100 times the rest, concentrated in the same channels across tokens. A single per-tensor INT8 scale set by the max is then dominated by these outliers, and everything else quantizes to a handful of levels, collapsing accuracy. LLM.int8() handles them by decomposition. The outlier columns (identified by a magnitude threshold) are computed in fp16, and the remaining 99.9% of the matrix in INT8, then summed. This is exact on the outliers and cheap on the bulk, and it is why 8-bit inference became usable for large models.

SmoothQuant (Xiao et al., 2022) attacks the same outliers differently, by migrating the difficulty from activations, which are hard to quantize, into weights, which are easy. Because a linear layer computes \( XW \), you can insert a per-channel diagonal rescaling \( X W = (X \diag(s)^{-1})(\diag(s) W) \) that leaves the product unchanged but shrinks the activation outliers by \( s_j \) and grows the corresponding weight rows. Choosing \( s_j = \max_i |X_{ij}|^{\alpha} / \max_i |W_{ji}|^{1-\alpha} \) balances the dynamic ranges so both sides quantize well, enabling full INT8 (weights and activations) with little accuracy loss.

GPTQ (Frantar et al., 2022) is a weight-only, one-shot, post-training quantizer that pushes to 3 to 4 bits by being careful about quantization order and compensation. It descends from Optimal Brain Quantization (Frantar and Alistarh, 2022), which frames rounding one weight as an error to be compensated by adjusting the not-yet-quantized weights. Minimizing the layerwise output error \( \| XW - X\hat W \|_2^2 \) makes the relevant curvature the Hessian \( H = 2 X^\top X \) (the input second-moment matrix). Quantizing weight \( w_q \) to \( \hat w_q \) and optimally compensating the rest updates the remaining weights by

the closed-form OBQ step, and then the row and column \( q \) are removed from \( H \) (a Gaussian elimination). GPTQ's contribution is to make this practical at LLM scale. It quantizes in a fixed column order, updates all remaining weights of the block with a single Cholesky-based solve rather than re-inverting the Hessian each step, and processes weights in blocks for cache efficiency. The result quantizes a 175B model to 3 to 4 bits in a few GPU-hours with small perplexity loss.

AWQ (Lin et al., 2023) starts from a different observation. Not all weights matter equally, and the ones that matter are identifiable from the activations. The weight channels that multiply large-magnitude activation features are salient. Protecting just the top ~1% of channels (by activation magnitude) from quantization error recovers most of the loss. Rather than keep those channels in high precision (which breaks hardware regularity), AWQ scales them up before quantizing and down after, per output channel, so the quantization grid spends more resolution where it matters. It is training-free, needs only a small calibration set to measure activation statistics, and produces 4-bit weights that are both accurate and fast to serve. GPTQ and AWQ are the two dominant weight-only 4-bit methods. They make similar accuracy at 4 bits, with AWQ often simpler to deploy and GPTQ more established.

FP8 vs INT8. INT8 is a uniform integer grid. FP8 (the E4M3 and E5M2 formats standardized by Micikevicius et al., 2022) spends some bits on an exponent, giving a floating grid that resolves small values finely and large values coarsely. For tensors with wide dynamic range (activations with outliers), FP8's non-uniform spacing tolerates the range without a decomposition trick, which is why H100-class hardware added native FP8 tensor cores and FP8 has become the default for high-throughput serving. INT8 remains competitive on weights and on hardware without FP8. The general accuracy/latency tradeoff is monotone but not linear. 8-bit is nearly lossless and roughly doubles compute throughput. 4-bit weight-only roughly halves weight bytes (a big decode-latency win) at a small, usually recoverable perplexity cost. Below 4 bits the accuracy cliff steepens and needs either mixed precision or quantization-aware training to stay usable.

KV-cache quantization. At long context and large batch, the KV cache, not the weights, dominates memory. For a model with \( L \) layers, \( n_{\text{kv}} \) key/value heads of dimension \( d_h \), the cache is \( 2 L\, n_{\text{kv}}\, d_h \) values per token per sequence. In fp16 a 32-layer model with 8 KV heads of dimension 128 costs \( 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072 \) bytes = 128 KB per token, so 8k tokens across a batch of 64 is 64 GB, exceeding the weights. Quantizing the cache to INT8 or FP8 halves that, directly doubling the batch or the context that fits, and the accuracy cost is small because attention is a weighted average that is forgiving of KV noise. This is why KV quantization is often the highest-leverage serving optimization. It trades a barely-measurable quality loss for a doubling of the throughput ceiling.

Distillation

Quantization shrinks a model's precision. Distillation shrinks its size by training a small student to imitate a large teacher. The original formulation (Hinton et al., 2015) trains the student to match the teacher's softened output distribution. With logits \( z \), the softened probability at temperature \( T \) is \( p_i^{(T)} = \softmax(z/T)_i \), and the student minimizes the KL to the teacher's softened distribution, usually alongside the ordinary hard-label loss,

The temperature is the point of the method. At \( T = 1 \) the teacher is nearly one-hot and conveys little beyond the label. Raising \( T \) softens the distribution and exposes the teacher's relative confidences over the wrong classes, the "dark knowledge" that class 7 looks a bit like class 1 but nothing like class 4, which is a far richer training signal than a hard label. The \( T^2 \) factor rescales the KL gradient, which shrinks like \( 1/T^2 \) under softening, so the distillation term keeps a stable magnitude as \( T \) changes. For language models the same idea takes several forms, matching next-token distributions (DistilBERT, Sanh et al., 2019, reaching ~97% of BERT quality at 40% of the size), matching hidden states, or, increasingly, sequence-level distillation where the teacher generates completions that become supervised targets for the student. The last of these is how much of today's frontier-to-small-model transfer actually happens. A strong model labels or generates data, a small model imitates it, and the small model inherits a slice of the large one's behavior at a fraction of the serving cost.

Retrieval-augmented generation

A language model's knowledge is frozen at its training cutoff and stored diffusely across its weights, where it cannot be edited, audited, or attributed. RAG (Lewis et al., 2020) separates the two kinds of knowledge. Parametric knowledge stays in the weights, and non-parametric knowledge goes into an external store that is retrieved at inference and placed in the context. The model then answers conditioned on retrieved passages rather than on memory alone, which lets it cite sources, stay current as the store is updated, and answer questions about private documents it never saw in pretraining. The pipeline has four stages, each with real design decisions.

Chunking. Documents are split into passages small enough to embed meaningfully and to fit several into a context budget, but large enough to be self-contained. Too small and a chunk loses the antecedent of a pronoun or the subject of a clause. Too large and its embedding averages several topics into a vague centroid that matches nothing well, and it wastes context budget. Typical sizes are a few hundred tokens with a small overlap (so a fact spanning a boundary appears whole in at least one chunk). Structure-aware chunking (split on headings, paragraphs, or sentences rather than a fixed token count) helps because it keeps semantic units intact. Problem 4 works the chunk-budget arithmetic. Given a context budget and a chunk size, it asks how many chunks you can retrieve and how that interacts with recall.

Embedding and retrieval. Each chunk is mapped to a dense vector by an embedding model, and the query is embedded the same way. Relevant chunks are the query vector's nearest neighbors under cosine similarity. Dense passage retrieval (Karpukhin et al., 2020) trains the encoder with a contrastive objective so that a question and its answer-bearing passage land close while unrelated passages are pushed apart. For a question \( q \), positive passage \( p^+ \), and negatives \( p^-_j \), minimize \( -\log \frac{e^{\text{sim}(q,p^+)}}{ e^{\text{sim}(q,p^+)} + \sum_j e^{\text{sim}(q,p^-_j)}} \). In-batch negatives make this cheap, since every other passage in the batch serves as a negative for free. DPR showed dense retrieval beating the strong lexical baseline BM25 on open-domain QA, and modern embedding models (trained on far more pairs, with hard negatives mined explicitly) are stronger still, though BM25 remains a hard-to-beat baseline for keyword-heavy and out-of-domain queries, so hybrid dense-plus-lexical retrieval is common. The nearest-neighbor search itself, over millions of vectors, uses an approximate index (HNSW, IVF), whose internals, graph construction, the recall/latency tradeoff, and when to prefer one over the other, are derived on the databases page. Here it is enough that the index returns an approximate top-\( N \) in sublinear time at a tunable recall.

Reranking with cross-encoders. The retriever is a bi-encoder. It embeds the query and each passage independently, so their vectors can be precomputed and indexed, which is what makes retrieval over millions of documents fast. The price of independence is that the query and passage never attend to each other, so the score is a blunt cosine between two summaries. A cross-encoder (Nogueira and Cho, 2019) concatenates the query and one candidate passage into a single sequence and runs full cross-attention, producing a far more accurate relevance score, at the cost of one forward pass per candidate, which cannot be precomputed. The standard architecture uses both. The bi-encoder retrieves a cheap top-\( N \) (say 100) from millions, and the cross-encoder reranks those \( N \) down to the top few that go into the prompt. This two-stage design is the retrieval analogue of a coarse filter followed by an exact check, and Problem 4 shows how reranking raises precision at fixed recall.

Evaluating RAG. RAG has two failure modes that ordinary generation metrics miss. The retriever can fail to surface the relevant passage, and the generator can ignore or contradict a passage it did retrieve. Evaluation therefore splits along the pipeline. Retrieval is scored with recall@\( k \) (did a relevant passage appear in the top \( k \)) and precision, or ranking metrics like MRR and nDCG. Generation is scored on faithfulness (groundedness). Does every claim in the answer follow from the retrieved context, or did the model hallucinate beyond it. Frameworks like RAGAS (Es et al., 2023) operationalize this by decomposing the answer into atomic claims and checking each against the retrieved context, usually with an LLM judge, and by measuring answer relevance and context precision separately. The key discipline is to attribute failures to the right stage. A low faithfulness score with high retrieval recall is a generation problem (the passage was there and the model ignored it). Low recall is a retrieval or chunking problem no prompt change will fix.

Tool use and agent loops

A model that can only emit text is limited to what it knows and can compute in a forward pass. Giving it tools, a calculator, a search API, a code interpreter, a database query, lets it offload what it is bad at (exact arithmetic, current facts, long computation) to systems that are good at it. The interface is function calling. The model is shown a set of tool schemas (name, description, typed arguments), and instead of answering it may emit a structured call naming a tool and its arguments. The runtime executes the call, appends the result to the context, and lets the model continue. The model never runs anything itself. It proposes calls that a trusted harness executes, which is also the security boundary.

ReAct (Yao et al., 2022) is the loop that makes this work for multi-step tasks. It interleaves reasoning traces with actions. The model thinks ("I need the population, then divide"), acts (calls a search tool), observes the result, thinks again, and repeats until it can answer. The reasoning steps keep the action selection grounded and let the model recover from a tool result that surprised it, and the actions keep the reasoning grounded in real observations rather than confabulated ones. The loop terminates when the model emits a final answer instead of a tool call, or when a step budget is hit. Reliability engineering on top of this, retries, verification, planning, self-correction, and the ways agents improve their own policies over episodes, is the subject of the self-improving agents page. Here the essential point is that the agent loop is a control loop around a stateless model, and its correctness depends far more on the harness (schema validation, error handling, tool sandboxing, budget enforcement) than on the model's raw capability.

Structured and constrained decoding

Function calling and any machine-consumed output need the model to emit text that parses, valid JSON, matching a schema, or conforming to a grammar. Prompting for JSON and hoping is unreliable. A single missing brace breaks the parser, and the failure rate is nonzero at any scale. Constrained decoding makes invalid output impossible by construction rather than unlikely by training. At each step the model produces a distribution over the vocabulary. Constrained decoding computes the set of tokens that could legally come next given the target grammar and the text generated so far, and masks the logits of every other token to \( -\infty \) before sampling. The model still chooses among valid continuations by its own probabilities, but it can never choose an invalid one.

The naive implementation, re-parsing the partial output against the grammar at every step to find allowed tokens, is too slow. It costs work proportional to the grammar and the output length on every one of thousands of decoding steps. The Outlines approach (Willard and Louf, 2023) precompiles the grammar (a regular expression, or a JSON schema lowered to one) into a finite-state machine, then builds an index mapping each FSM state to the set of vocabulary tokens that advance it to a valid next state. At decode time, generating a token is a single lookup. From the current FSM state, read the precomputed allowed-token mask, apply it, sample, and transition. The per-step cost drops from a parse to a dictionary lookup, making constrained generation nearly free relative to unconstrained. The construction is done once per schema and amortized over every request that uses it. The demo in the implementation section builds a tiny version of this mask. Grammars richer than regular (full context-free JSON with arbitrary nesting) need a pushdown automaton rather than a finite one, which is what production libraries implement, but the finite-state case carries the whole idea.

Evaluation harnesses and LLM-as-judge

Multiple-choice benchmarks with a known answer are scored automatically by a harness that formats each question, runs the model, and extracts the answer, and the main correctness subtlety is scoring. Comparing the log-likelihood the model assigns to each answer choice is more robust than parsing free-form generation, and small formatting differences (how options are labeled, whether a space precedes the answer) shift scores by points, which is why a standardized harness like lm-evaluation-harness exists so numbers are comparable across models. Open-ended quality, whether a summary is good, whether a chatbot response is helpful, has no reference answer, and human evaluation is slow and expensive. The now-standard shortcut is LLM-as-judge, prompting a strong model to score or compare responses. MT-Bench and the Chatbot Arena analysis (Zheng et al., 2023) validated that a strong judge model agrees with human preferences at roughly the rate two humans agree with each other, which makes it a usable proxy.

The biases are real and must be controlled. Judges show position bias (a systematic preference for the first or second response in a pairwise comparison, independent of content), verbosity bias (longer answers rated higher even when padded), self-preference (a judge favors text in its own style, or its own family's outputs), and sensitivity to superficial fluency over correctness. The mitigations are mechanical. Randomize and average over both orderings to cancel position bias, control for length, use multiple judges or a reference answer, and reserve human evaluation for the high-stakes final comparison. The honest framing is that LLM-as-judge is a fast, cheap, biased instrument, excellent for iteration and regression testing, dangerous as the sole basis of a headline claim, and never a substitute for a task-grounded metric when one exists.

The serving cost model

The economics of a deployed model come down to tokens per second per dollar, and the number that governs it is which resource is the bottleneck. Generation has two phases with opposite characteristics. Prefill processes the whole prompt in one forward pass. All prompt tokens go through the network together, so the matmuls are large and dense and the phase is compute-bound, running near the hardware's peak FLOP rate. Decode generates one token at a time, each conditioned on all previous tokens via the KV cache. Each step is a single-token forward pass, a sequence of matrix-vector products, and to produce one token the hardware must read every weight from HBM. Decode is therefore memory-bandwidth-bound at small batch. Its speed is set not by how fast the GPU multiplies but by how fast it can stream the weights.

Put numbers on it with the measured H100 80GB figures from this repository. Sustained memory bandwidth for a bf16 streaming kernel is 2930 GB/s (the copy_bf16 measurement), and sustained bf16 matmul throughput at large size is 728.7 TFLOP/s (the n8192 measurement). For a 7B model in fp16, decoding one token for a single sequence reads all \( 2 \times 7\times 10^9 = 1.4 \times 10^{10} \) weight bytes, so the batch-1 decode ceiling is

a hard roofline no kernel cleverness beats, because it is set by bandwidth and model size alone. The way out is continuous batching (Yu et al., 2022, and the vLLM system, Kwon et al., 2023), serving many sequences at once so that a single read of the weights produces a token for every sequence in the batch. The weights are read once per decode step regardless of batch size \( B \), while the arithmetic scales with \( B \), so decode stays memory-bound until \( B \) is large enough that the compute of \( B \) tokens takes longer than the single weight read. That crossover, the roofline ridge point, is the ratio of peak compute to peak bandwidth,

in units of FLOP per byte. Below \( B^* \) adding sequences is nearly free (same weight read, more tokens out). Above it the GPU is compute-bound and throughput saturates at \( 728.7\times 10^{12} / (2 \times 7\times 10^9) \approx 52{,}000 \) tokens/s aggregate. This factor-of-250 spread between batch-1 latency-optimal serving and batch-saturated throughput-optimal serving is the central fact of LLM serving economics, and it is why the cost per token depends far more on how you batch than on which GPU you rent. Problem 3 turns these throughputs into dollars per million tokens, roughly $3.98 per million at batch 1 versus $0.016 per million at saturation, from the identical hardware. Continuous batching, which admits and evicts sequences from the batch as they arrive and finish rather than waiting for a fixed batch to drain, plus paged KV cache (Kwon et al., 2023) so the cache is not fragmented into fixed slots, is what lets a real server run near the throughput ceiling while still bounding per-request latency.

Worked problems

Implementation

The first block implements a LoRA linear layer from scratch in PyTorch and JAX, a frozen base matmul plus the scaled low-rank path, with \( B \) initialized to zero so the adapted layer starts identical to the base. The shapes are annotated. Note that only \( A \) and \( B \) carry gradients.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    # y = x W0^T + (alpha/r) x A^T B^T ,  W0 frozen
    def __init__(self, d_in, d_out, r=16, alpha=32):
        super().__init__()
        self.W0 = nn.Linear(d_in, d_out, bias=False)
        self.W0.weight.requires_grad_(False)        # base is frozen
        self.A = nn.Parameter(torch.randn(r, d_in) / (r ** 0.5))  # (r, d_in)
        self.B = nn.Parameter(torch.zeros(d_out, r))              # (d_out, r), starts at 0
        self.scale = alpha / r

    def forward(self, x):                # x: (batch, d_in)
        base = self.W0(x)                # (batch, d_out), frozen path
        lora = (x @ self.A.T) @ self.B.T # (batch, r) then (batch, d_out)
        return base + self.scale * lora

# sanity: at init B=0 so the layer equals its frozen base
layer = LoRALinear(4096, 4096, r=16, alpha=32)
x = torch.randn(8, 4096)
assert torch.allclose(layer(x), layer.W0(x), atol=1e-5)
trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
print("trainable params:", trainable)   # 16*(4096+4096) = 131072

import jax, jax.numpy as jnp
from flax import linen as nn

class LoRALinear(nn.Module):
    d_out: int
    r: int = 16
    alpha: float = 32.0

    @nn.compact
    def __call__(self, x):               # x: (batch, d_in)
        d_in = x.shape[-1]
        # frozen base: mark non-trainable by stopping its gradient
        W0 = self.param("W0", nn.initializers.lecun_normal(), (d_in, self.d_out))
        W0 = jax.lax.stop_gradient(W0)
        A = self.param("A", lambda k, s: jax.random.normal(k, s) / (self.r ** 0.5),
                       (self.r, d_in))    # (r, d_in)
        B = self.param("B", nn.initializers.zeros, (self.d_out, self.r))  # (d_out, r)
        base = x @ W0                      # (batch, d_out)
        lora = (x @ A.T) @ B.T             # (batch, r) then (batch, d_out)
        return base + (self.alpha / self.r) * lora

key = jax.random.PRNGKey(0)
model = LoRALinear(d_out=4096, r=16, alpha=32.0)
x = jax.random.normal(key, (8, 4096))
params = model.init(key, x)
# at init B=0, so output equals the frozen base path
base = x @ jax.lax.stop_gradient(params["params"]["W0"])
assert jnp.allclose(model.apply(params, x), base, atol=1e-4)

The second block implements the DPO loss in both frameworks. The inputs are per-sequence summed log-probabilities of the chosen and rejected completions under the trainable policy and the frozen reference. The loss is the negative log-sigmoid of the \( \beta \)-scaled difference of implicit rewards, exactly the closed form derived above.

import torch
import torch.nn.functional as F

def dpo_loss(logp_w, logp_l, ref_logp_w, ref_logp_l, beta=0.1):
    # all args: (batch,) summed log-probs of the full completion
    rhat_w = beta * (logp_w - ref_logp_w)   # implicit reward, chosen
    rhat_l = beta * (logp_l - ref_logp_l)   # implicit reward, rejected
    margin = rhat_w - rhat_l                 # (batch,)
    loss = -F.logsigmoid(margin).mean()      # -log sigma(margin)
    acc = (margin > 0).float().mean()        # fraction ranked correctly
    return loss, acc

# reproduce Problem 2
logp_w = torch.tensor([-12.0]); ref_w = torch.tensor([-13.0])
logp_l = torch.tensor([-15.0]); ref_l = torch.tensor([-14.0])
loss, acc = dpo_loss(logp_w, logp_l, ref_w, ref_l, beta=0.1)
print(float(loss), float(acc))   # ~0.598, 1.0

import jax.numpy as jnp
import jax.nn as jnn

def dpo_loss(logp_w, logp_l, ref_logp_w, ref_logp_l, beta=0.1):
    # all args: (batch,) summed log-probs of the full completion
    rhat_w = beta * (logp_w - ref_logp_w)    # implicit reward, chosen
    rhat_l = beta * (logp_l - ref_logp_l)    # implicit reward, rejected
    margin = rhat_w - rhat_l
    loss = -jnp.mean(jnn.log_sigmoid(margin))
    acc = jnp.mean((margin > 0).astype(jnp.float32))
    return loss, acc

logp_w = jnp.array([-12.0]); ref_w = jnp.array([-13.0])
logp_l = jnp.array([-15.0]); ref_l = jnp.array([-14.0])
loss, acc = dpo_loss(logp_w, logp_l, ref_w, ref_l, beta=0.1)
print(float(loss), float(acc))   # ~0.598, 1.0

The third block is a minimal end-to-end RAG pipeline in NumPy. It embeds a small corpus, retrieves the top-\( k \) by cosine similarity, and assembles a grounded prompt. It uses a toy hashing embedder so it runs with no downloads. In production the embedder is a trained dense encoder and the linear scan is replaced by an approximate index (see the databases page).

import numpy as np

def embed(text, dim=256):
    # toy bag-of-hashed-tokens embedder; stands in for a trained encoder
    v = np.zeros(dim)
    for tok in text.lower().split():
        v[hash(tok) % dim] += 1.0
    n = np.linalg.norm(v)
    return v / n if n > 0 else v

corpus = [
    "The H100 GPU has 80 GB of HBM3 memory.",
    "LoRA freezes the base model and trains low-rank adapters.",
    "Continuous batching amortizes weight reads across many sequences.",
    "Paris is the capital of France.",
]
emb = np.stack([embed(c) for c in corpus])       # (n_docs, dim)

def retrieve(query, k=2):
    q = embed(query)                              # (dim,)
    sims = emb @ q                                # (n_docs,) cosine, since normalized
    topk = np.argsort(-sims)[:k]                  # indices of k best
    return [(corpus[i], float(sims[i])) for i in topk]

def build_prompt(query, k=2):
    hits = retrieve(query, k)
    context = "\n".join(f"[{i+1}] {doc}" for i, (doc, _) in enumerate(hits))
    return f"Context:\n{context}\n\nQuestion: {query}\nAnswer using only the context."

print(retrieve("how much memory does the H100 have", k=2))
print(build_prompt("what does LoRA train", k=2))

The fourth block demonstrates constrained decoding, with a finite-state machine for a small grammar (a signed integer, then a decimal point and digits) and a per-step logit mask that zeroes out any token whose character cannot legally follow the text produced so far. This is the finite-state kernel of the Outlines approach, on a character vocabulary for clarity.

import numpy as np

# Grammar: optional '-', one or more digits, optional ('.' one or more digits).
# States: START, INT (in integer part), DOT (just saw '.'), FRAC (in fraction).
VOCAB = list("0123456789.-")             # character vocabulary
DIGITS = set("0123456789")

def allowed(state, seen_frac_digit):
    # return the set of characters that keep the string grammar-valid
    if state == "START":
        return DIGITS | {"-"}
    if state == "INT":
        return DIGITS | {"."}            # more digits, or start a fraction
    if state == "DOT":
        return DIGITS                    # must have at least one fraction digit
    if state == "FRAC":
        return DIGITS                    # more fraction digits only
    return set()

def step(state, ch):
    if state in ("START", "INT"):
        if ch in DIGITS: return "INT"
        if ch == "-":    return "INT" if state == "START" else state
        if ch == "." and state == "INT": return "DOT"
    if state in ("DOT", "FRAC") and ch in DIGITS:
        return "FRAC"
    raise ValueError("illegal transition")

def mask_logits(logits, state):
    # set logits of disallowed tokens to -inf so sampling can never pick them
    ok = allowed(state, None)
    masked = logits.copy()
    for i, ch in enumerate(VOCAB):
        if ch not in ok:
            masked[i] = -np.inf
    return masked

# greedy-decode a grammar-valid number from arbitrary raw logits
rng = np.random.default_rng(0)
state, out = "START", ""
for _ in range(6):
    logits = rng.normal(size=len(VOCAB))
    m = mask_logits(logits, state)
    ch = VOCAB[int(np.argmax(m))]        # best *legal* token
    out += ch
    state = step(state, ch)
print(out)   # always a valid signed number, e.g. "-3.72" or "580"

How it is done in practice

The gap between these derivations and a deployed system is mostly plumbing, and the plumbing is where the reliability lives. Fine-tuning in practice is almost never full fine-tuning. The default is LoRA or QLoRA through a library like Hugging Face PEFT, with the base model quantized by bitsandbytes and the training loop wrapped by TRL for the DPO or SFT objective. The practitioner's decisions are which layers to adapt (attention projections are cheapest, while adapting the MLP too costs more but helps on harder shifts), the rank (8 to 64 covers most cases, and higher ranks rarely help and risk overfitting small datasets), and the learning rate (LoRA tolerates larger learning rates than full fine-tuning because the update is constrained). A common failure is forgetting that a merged adapter bakes in the base's dtype. Merge a QLoRA adapter into a 4-bit base and the merge is lossy, so people merge into a dequantized base and re-quantize.

Serving in practice means vLLM, TensorRT-LLM, or a similar engine, not a bare model forward. The engine implements continuous batching and paged KV cache so that the 250-fold throughput spread from Problem 3 is actually captured, and it applies a serving quantization (FP8 or 4-bit weight-only via GPTQ/AWQ) chosen against the accuracy budget. The KV cache is usually the binding constraint at scale, so paged attention (which stores the cache in fixed-size blocks like OS pages, eliminating the fragmentation that fixed per-sequence slots cause) and KV quantization together set the achievable batch size, and therefore the cost. The prefill/decode split shows up as a real scheduling problem. Prefill is a compute-heavy burst that can stall ongoing decodes, so engines chunk long prefills and interleave them with decode steps (chunked prefill) to keep tail latency bounded. Speculative decoding, a small draft model proposes several tokens that the large model verifies in one batched forward pass, is the main lever for cutting decode latency below the batch-1 roofline, because it turns several sequential memory-bound steps into one, at the cost of some wasted draft compute.

RAG in practice is dominated by the unglamorous parts, such as chunking strategy, keeping the index fresh as documents change, and hybrid retrieval (dense plus BM25) because pure dense retrieval misses exact-match queries like error codes and part numbers. The reranker is often the highest- value single addition, because retrieval recall is usually fine and precision is what fails, as Problem 4 showed. Evaluation is run continuously. A held-out set of question-answer-source triples measures retrieval recall and answer faithfulness on every index or prompt change, so that a regression is caught before it ships. And the agent layer, tool calling in a loop, needs hard engineering around the model. That means schema validation on every tool call, timeouts and retries, sandboxed execution, and a strict step budget, because an unbounded agent loop is a way to spend money and hit rate limits with nothing to show. Constrained decoding is what makes the tool-call JSON parse every time rather than almost every time, which is the difference between a demo and a service.

The current research frontier

Parameter-efficient adaptation keeps getting sharper. DoRA (Liu et al., NVIDIA, 2024) decomposes the weight update into magnitude and direction and applies LoRA only to the direction, closing much of the remaining gap to full fine-tuning. VeRA (Kopiczko et al., Amsterdam, 2024) shares frozen random \( A \) and \( B \) across layers and trains only tiny per-layer scaling vectors, cutting adapter parameters by another order of magnitude. On the quantization side, the frontier is sub-4-bit and quantization-aware pretraining. AQLM (Egiazarian et al., 2024) uses additive vector quantization to reach 2 to 3 bits with surprising quality, QuIP# (Cornell, 2024) uses incoherence processing and lattice codebooks, and the "1.58-bit" BitNet line (Microsoft Research, 2024) trains ternary-weight models from scratch, arguing that quantization belongs in pretraining rather than as a post-hoc step.

Alignment beyond DPO is a crowded field. The direct-preference family has spawned IPO (Azar et al., DeepMind, 2023), which fixes DPO's tendency to overfit deterministic preferences, KTO (Ethayarajh et al., 2024), which learns from unpaired good/bad labels using a prospect-theory value function so it does not need preference pairs, and ORPO (Hong et al., 2024), which folds preference optimization into the SFT stage with no reference model at all. Meanwhile online and on-policy variants argue that DPO's offline nature is the real limitation and that a short RL phase, or on-policy preference sampling, recovers what DPO gives up. GRPO (DeepSeek, 2024) and the reasoning-model training that made it famous are treated on the deep RL page. Retrieval is moving toward learned, end-to-end pipelines, with late-interaction retrievers like ColBERT (Khattab and Zaharia, Stanford, 2020) that keep per-token embeddings for finer matching, and retrieval that is trained jointly with the generator rather than bolted on. Long-context models raise the question of whether RAG is even needed. The emerging answer is that retrieval and long context are complementary, since a 100k-token window still cannot hold a corporate wiki, and retrieval is cheaper than attending over everything. On serving, the research is in scheduling (disaggregated prefill and decode on separate hardware pools, as in DistServe and Splitwise) and in speculative decoding variants (Medusa's multiple heads, EAGLE's feature-level drafting) that push decode throughput past the single-model roofline.

Open source to read

• huggingface/peft, the reference implementation of LoRA, DoRA, prefix tuning, and the adapter family. Read src/peft/tuners/lora/layer.py to see the exact forward-pass reparametrization and the merge logic.
• huggingface/trl, which provides SFT, reward modeling, PPO, and DPO trainers. Open trl/trainer/dpo_trainer.py and find the concatenated-forward that computes chosen and rejected log-probs in one pass, and the loss that matches the derivation above.
• artidoro/qlora, the original QLoRA code, small enough to read end to end. qlora.py shows NF4 loading, the paged optimizer, and the training loop.
• TimDettmers/bitsandbytes, the 4-bit and 8-bit kernels underneath QLoRA and LLM.int8(). The NF4 quantization and blockwise double-quantization live in the CUDA sources under csrc/.
• vllm-project/vllm, production serving with paged attention and continuous batching. Start with the paged KV-cache block manager and the scheduler to see how the throughput ceiling from Problem 3 is captured.
• outlines-dev/outlines, constrained decoding via FSM compilation. Read the regex-to-FSM index construction to see how the per-step allowed-token mask is precomputed and cached.
• run-llama/llama_index, a full RAG toolkit, good for seeing how chunking, indexing, retrieval, and reranking are composed into a pipeline, and where the seams are.
• EleutherAI/lm-evaluation-harness, the standard benchmark harness. Read a task YAML and the log-likelihood scoring path to see why formatting choices move scores.

Common misconceptions

"LoRA saves memory because it trains fewer parameters." It saves memory because it trains fewer parameters and each trainable parameter carries 16 bytes of optimizer state while each frozen one carries 2. The dominant term is the optimizer state (three fp32 tensors per trainable parameter), not the weights, which is why the saving is a factor of hundreds, not the factor implied by parameter count alone. Problem 1 makes this explicit, with 112 GB of state collapsing to 134 MB.

"QLoRA quantizes the model, so it must hurt quality." QLoRA quantizes the frozen base to 4 bits during fine-tuning, but the adapters train in bf16 and the gradients flow through dequantized weights. The NF4 data type is matched to the Gaussian shape of weights, and the original paper matched 16-bit LoRA quality. The base's precision is a storage decision, not a capability ceiling.

"DPO is just supervised fine-tuning on the chosen responses." It is not. Cloning the chosen response has no term that pushes the rejected one down, and no reference model. DPO's gradient contains the difference \( \nabla \log \pi_\theta(y_w) - \nabla \log \pi_\theta(y_l) \) weighted by a sigmoid that self-anneals as the pair is learned. It optimizes a relative preference against a frozen reference, which is why it changes behavior that plain SFT on the same chosen text does not.

"A bigger GPU makes decoding faster." Batch-1 decode is bounded by memory bandwidth and model size, not FLOP throughput. A GPU with twice the FLOPs but the same bandwidth decodes a single stream no faster. The way to use more compute is to batch more sequences, up to the ridge point near \( B^* \approx 249 \). Throughput and per-stream latency are different axes.

"RAG fixes hallucination." RAG grounds answers when the relevant passage is retrieved and the model uses it. It introduces two new failure modes, the retriever missing the passage (a recall problem no prompt fixes) and the generator ignoring or contradicting a retrieved passage (a faithfulness problem). Measuring these separately, as in Problem 4 and the RAG-evaluation discussion, is the only way to know which one is biting.

"Constrained decoding makes the model smarter." It makes the output parse, not the content correct. Masking illegal tokens guarantees valid JSON or a grammar-conformant string, but the model can still fill valid structure with wrong values. Constrained decoding buys reliability of form, which is a real and necessary property for tool-calling, and nothing about the substance.

"LLM-as-judge is objective because it is automated." It has measurable position bias, verbosity bias, and self-preference. It is a fast, cheap, biased instrument, useful for iteration and regression testing, and it must be debiased (order randomization, length control, multiple judges) and backstopped by human evaluation for any decision that matters.

Self-check

References

1. Jurafsky, D. and Martin, J. H. Speech and Language Processing, 3rd edition draft. Stanford, ongoing. The standard reference for NLP fundamentals underlying retrieval and generation.
2. Murphy, K. P. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. Covers the variational and information-theoretic tools behind KL-regularized objectives and quantization.
3. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. 2021. arXiv:2106.09685.
4. Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. 2020. arXiv:2012.13255.
5. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. 2023. arXiv:2305.14314.
6. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. 2022. arXiv:2208.07339.
7. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. 2022. arXiv:2210.17323.
8. Frantar, E. and Alistarh, D. Optimal Brain Compression / Optimal Brain Quantization. 2022. arXiv:2208.11580.
9. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. 2023. arXiv:2306.00978.
10. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. 2022. arXiv:2211.10438.
11. Micikevicius, P., et al. FP8 Formats for Deep Learning. 2022. arXiv:2209.05433.
12. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. 2023. arXiv:2305.18290.
13. Ouyang, L., et al. Training Language Models to Follow Instructions with Human Feedback (InstructGPT). 2022. arXiv:2203.02155.
14. Wei, J., et al. Finetuned Language Models Are Zero-Shot Learners (FLAN). 2021. arXiv:2109.01652.
15. Zhou, C., et al. LIMA: Less Is More for Alignment. 2023. arXiv:2305.11206.
16. Houlsby, N., et al. Parameter-Efficient Transfer Learning for NLP (Adapters). 2019. arXiv:1902.00751.
17. Li, X. L. and Liang, P. Prefix-Tuning: Optimizing Continuous Prompts for Generation. 2021. arXiv:2101.00190.
18. Hinton, G., Vinyals, O., and Dean, J. Distilling the Knowledge in a Neural Network. 2015. arXiv:1503.02531.
19. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. DistilBERT, a distilled version of BERT. 2019. arXiv:1910.01108.
20. Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. 2020. arXiv:2005.11401.
21. Karpukhin, V., et al. Dense Passage Retrieval for Open-Domain Question Answering. 2020. arXiv:2004.04906.
22. Izacard, G. and Grave, E. Leveraging Passage Retrieval with Generative Models for Open Domain QA (Fusion-in-Decoder). 2021. arXiv:2007.01282.
23. Nogueira, R. and Cho, K. Passage Re-ranking with BERT. 2019. arXiv:1901.04085.
24. Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. RAGAS: Automated Evaluation of Retrieval Augmented Generation. 2023. arXiv:2309.15217.
25. Yao, S., et al. ReAct: Synergizing Reasoning and Acting in Language Models. 2022. arXiv:2210.03629.
26. Willard, B. T. and Louf, R. Efficient Guided Generation for Large Language Models (Outlines). 2023. arXiv:2307.09702.
27. Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. 2023. arXiv:2306.05685.
28. Kwon, W., et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). 2023. arXiv:2309.06180.
29. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. 2019. arXiv:1910.02054.
30. Liu, S.-Y., et al. DoRA: Weight-Decomposed Low-Rank Adaptation. 2024. arXiv:2402.09353.