Long-form write-ups of the coursework I have worked through, rebuilt around what the field looks like now rather than what a syllabus covered when I sat in the room. Each page states the problem, derives the math step by step in full notation, works representative problems by hand, and implements the result in PyTorch and JAX side by side, with C++, Rust, and CUDA where the subject calls for it. Where a claim is about performance, the number comes from a benchmark run on an H100 in this repository rather than from memory. The references are textbooks and papers, and every page ends with the open-source repositories worth reading and a short self-check quiz.
The full pipeline with nothing hidden behind a library: BPE implemented and analyzed, a transformer with RoPE, SwiGLU, RMSNorm, and grouped-query attention, FlashAttention's tiling derived, data-parallel/tensor-parallel/pipeline sharding and FSDP, scaling laws and compute-optimal budgeting, mixture-of-experts routing, and an inference path with paged KV cache and speculative decoding.
Read the write-up →The mathematical core: what a learning algorithm is, why it generalizes, how to fit one, and how to reason under uncertainty when the model is a graph of random variables rather than a single predictor.
Supervised learning derived end to end: the normal equations and their geometry, logistic regression and Newton's method, generalized linear models from the exponential family, generative vs discriminative classifiers, kernels and the representer theorem, the bias-variance decomposition, and EM derived as a lower-bound maximization.
The pre-deep-learning half of AI that still decides real systems: uniform-cost and A* with admissible heuristics proved optimal, minimax with alpha-beta and expectimax, MDP value and policy iteration, constraint satisfaction with arc consistency, Bayesian networks and particle filtering, and where each one shows up inside a modern agent stack.
Concentration inequalities built from scratch, PAC learning and VC dimension, Rademacher complexity, margin bounds that explain boosting and SVMs, algorithmic stability, the implicit bias of gradient descent, and the double-descent picture that broke the classical story.
The practical mechanics of getting a network to converge: signal propagation and He/Xavier initialization derived from variance analysis, batch/layer/RMS normalization compared, residual streams, optimizer and schedule choice, regularization, and a systematic error-analysis loop for deciding what to fix next.
Bayesian networks and Markov random fields, d-separation and the independence semantics, exact inference by variable elimination and the junction tree, loopy belief propagation as a variational method, MCMC and variational inference, structure learning, and the line from these ideas to modern latent-variable models.
How the modern generative stack is actually built: tokenizers and attention up through pretraining at scale, diffusion, alignment, and the multimodal models that share one architecture across text, image, audio, and graphs.
The full pipeline with nothing hidden behind a library: BPE implemented and analyzed, a transformer with RoPE, SwiGLU, RMSNorm, and grouped-query attention, FlashAttention's tiling derived, data-parallel/tensor-parallel/pipeline sharding and FSDP, scaling laws and compute-optimal budgeting, mixture-of-experts routing, and an inference path with paged KV cache and speculative decoding.
Word2vec and GloVe derived with their gradients, recurrent models and the vanishing-gradient analysis that motivated LSTMs, seq2seq and the attention mechanism that replaced it, subword tokenization, BERT-style masked pretraining vs autoregressive pretraining, and the fine-tuning and prompting regimes that followed.
Contextual representations and what probing does and does not show, natural language inference and the annotation artifacts that inflate it, retrieval-augmented and open-domain question answering, compositional generalization, adversarial and contrast sets, and how to build a benchmark whose numbers survive contact with a new model.
From the waveform up: framing, the STFT, and mel filterbanks derived; HMM-GMM recognition as the historical baseline; CTC with its forward algorithm and gradient worked out; RNN-transducer for streaming; Whisper-style encoder-decoder ASR; speaker diarization; neural audio codecs; and text-to-speech from Tacotron to diffusion vocoders.
Why quadratic attention becomes the binding constraint, the HiPPO initialization and the S4 kernel derived, Mamba's selective scan and its hardware-aware parallel form, linear attention as an RNN in disguise, RWKV and RetNet, hybrid architectures, and the associative recall tasks where each design actually breaks.
One framework for the whole family: the ELBO derived twice for VAEs, normalizing flows and the change-of-variables formula, autoregressive likelihood models, energy-based models with contrastive divergence, score matching and the connection between denoising score matching and diffusion, plus how each family trades off likelihood, sample quality, and sampling speed.
The minimax objective and its optimal discriminator, why the original loss saturates and what the non-saturating and Wasserstein variants fix, gradient penalties and spectral normalization, progressive and style-based generators, conditional and cycle-consistent translation, FID and its failure modes, and where GANs still beat diffusion.
Diffusion derived from both the variational and score-based views and shown to be the same model, DDPM and DDIM sampling, classifier-free guidance, the probability-flow ODE and its solvers, latent diffusion, flow matching and rectified flow, consistency and distillation methods for few-step sampling, diffusion transformers, and controllable generation.
CLIP's contrastive objective and its temperature, cross-attention vs early fusion vs adapter bridging, vision-language models from Flamingo to native multimodal transformers, audio and video encoders, unified tokenization across modalities, alignment and hallucination in multimodal settings, and how to evaluate a model that reads and draws.
Turning a pretrained model into a product: parameter-efficient fine-tuning with LoRA and QLoRA derived, instruction tuning, RLHF and DPO, quantization and distillation for serving, retrieval pipelines with chunking and reranking, tool use and agent loops, structured decoding, evaluation harnesses, and the cost model behind each choice.
How strong corpora are actually built and how model quality is actually measured. Gathering and cleaning at scale with extraction, language identification, quality filtering, and MinHash deduplication with the banding math derived, then LLM-as-a-judge used carefully with its documented biases and a kappa calibration workflow, and finally when one sample is enough versus when majority voting, pass@k, and confidence intervals are the honest answer.
A survey of the whole audio stack as it stands now: the classic DSP-and-HMM pipeline versus learned features and neural codecs, the model families from Whisper and wav2vec through VALL-E, VITS, and Moshi to gpt-realtime and full-duplex interaction models, what is known about model sizes, how to fine-tune ASR and TTS, and an honest comparison of the labs and products, from ElevenLabs and Cartesia to the orchestration layers like Vapi that wire them into agents.
Node embeddings from random walks, the message-passing framework with GCN, GraphSAGE, and GAT derived, the Weisfeiler-Lehman expressiveness bound and what it forbids, over-smoothing and over-squashing, graph transformers with positional encodings, heterogeneous and knowledge graphs, sampling for billion-edge graphs, and molecular and recommendation applications.
Acting under uncertainty: the theory of MDPs, the deep RL algorithms that made it work on pixels and robots, and the meta-learning and self-improvement loops now driving language model post-training.
Markov decision processes and the Bellman equations, contraction mapping proofs for value and policy iteration, Monte Carlo vs temporal-difference learning, function approximation and the deadly triad, exploration from epsilon-greedy through UCB and posterior sampling with their regret bounds, batch and offline RL, and policy gradient theory.
REINFORCE derived from the likelihood ratio with baselines and GAE, TRPO's trust region and PPO's clipped surrogate, DQN and its stabilizers, DDPG/TD3/SAC and the maximum-entropy framework, model-based RL and Dreamer, offline RL with CQL and IQL, imitation and inverse RL, and RLHF/GRPO as the same algorithms applied to token-level MDPs.
Options and hierarchical RL with the semi-MDP framework, successor features and transfer, multi-agent learning and equilibrium concepts, self-play and population training, distributional RL, exploration by intrinsic motivation, constrained MDPs and safe RL, and the reproducibility problems that plague the field's benchmarks.
Utility theory and decision networks, exact and approximate POMDP solutions from alpha vectors to POMCP, belief updates and filtering, Monte Carlo tree search with UCT derived, value of information, Gaussian processes and Bayesian optimization for expensive decisions, and validation of safety-critical policies.
Multi-task architectures and the gradient-conflict problem, MAML's second-order gradients derived with first-order approximations, prototypical and metric-based few-shot learning, black-box and in-context meta-learners, task distributions and the meta-overfitting trap, continual learning and catastrophic forgetting, and why in-context learning in large models is meta-learning that emerged for free.
Chain-of-thought as a latent variable, self-consistency and best-of-n with their scaling behavior, process vs outcome reward models, rejection sampling and expert iteration, RL with verifiable rewards, self-critique and debate, memory and skill libraries, tool-augmented reasoning, test-time compute scaling laws, and the reward-hacking failure modes each loop invites.
Game description as logic and the general-game-playing problem, minimax with alpha-beta and modern search enhancements, Monte Carlo tree search from bandits to UCT, AlphaZero's policy-value network and self-play loop derived, MuZero's learned model, imperfect-information games and counterfactual regret minimization, and what transfers from games to real decision problems.
Where performance actually comes from: the machine underneath, the concurrency model on top of it, and the data systems that make terabytes tractable.
Built bottom up: digital logic and timing, the single-cycle and pipelined datapath with hazards and forwarding, branch prediction, caches and the full memory hierarchy with worked AMAT arithmetic, virtual memory and TLBs, out-of-order execution with Tomasulo, superscalar and speculation, cache coherence and memory consistency models, SIMD, and the GPU as a throughput machine, ending at systolic-array accelerators and the roofline model.
Modern DRAM and HBM internals, prefetching and memory-level parallelism, non-uniform memory access, interconnects from PCIe to NVLink, tensor cores and systolic arrays with their dataflow taxonomy, sparsity and quantization support in hardware, performance counters and how to read a profile, and measured H100 roofline numbers from this machine.
Parallel decomposition and the cost of communication, Amdahl and Gustafson worked numerically, SPMD and ISPC-style vectorization, work-efficient parallel algorithms (scan, reduce, sort), the CUDA execution model with warps, occupancy, and coalescing, shared-memory tiling for matmul, warp shuffles and reductions, Triton kernels, attention kernel design, and measured bandwidth and TFLOPS from an H100 on this machine.
One concurrency curriculum written four times: processes and the fork/exec/wait model, POSIX threads and C++ std::thread, Rust's Send/Sync and how the borrow checker eliminates data races at compile time, atomics and the acquire/release memory model, lock-free queues, Python's GIL and the free-threaded build, async/await and event loops, and measured throughput comparisons across all four.
The kernel's core abstractions built up: process and thread implementation, context switching and scheduling from round-robin to CFS and EEVDF, synchronization primitives implemented from atomics, virtual memory with page tables and replacement policies, file systems and journaling, I/O and the block layer, containers as namespaces and cgroups, and virtualization.
The layered model with the arguments for and against it, reliable delivery and TCP's congestion control derived including the throughput equation, BBR's model-based alternative, queueing and active queue management, datacenter topologies and load balancing, RDMA and collective communication for distributed training, QUIC and HTTP/3, and software-defined networking.
The relational model and algebra with SQL as its surface, storage layouts row vs column, B-trees and LSM trees compared with their write amplification math, join algorithms and cost-based query optimization worked through an example plan, transactions and isolation levels with the anomalies each permits, ARIES recovery, multi-version concurrency control, distributed consensus and sharding, and vector search as a first-class index type.
Algorithms for data that does not fit in memory: MapReduce and its cost model, similarity search with minhash and locality-sensitive hashing derived with the S-curve, frequent itemsets and A-Priori, streaming with Bloom filters, Count-Min, HyperLogLog, and reservoir sampling (with the error bounds proved), PageRank and random walks with teleport, recommendation systems and matrix factorization, submodular maximization, and large-scale clustering.
Front end through back end: parsing, intermediate representations and SSA construction, dataflow analysis as a lattice fixed point with worked iterations, loop transformations including tiling and fusion, register allocation by graph coloring, instruction scheduling, polyhedral analysis, and how XLA, TorchInductor, and Triton apply all of it to tensor programs.
Operational semantics and the progress/preservation proof, lambda calculus and type inference with Hindley-Milner unification worked by hand, polymorphism and subtyping, algebraic data types and pattern matching, effects and monads, linear and affine types as the theory behind Rust's ownership, gradual typing, and memory-safety guarantees compared across languages.
The analytical core: how to design an algorithm and prove it works, how to optimize a function or a combinatorial structure, and how to build systems that stay correct against an adversary.
The design paradigms with their proof techniques: asymptotic analysis and the master theorem applied, divide and conquer with recurrences solved, greedy algorithms proved correct by exchange arguments, dynamic programming derived from optimal substructure, graph algorithms including shortest paths and minimum spanning trees, maximum flow with min-cut duality, and NP-completeness with reductions worked in full.
Structures worth knowing past the interview set: amortized analysis by potential functions, balanced trees and splay trees with their access-time proof, Fibonacci heaps, union-find with the inverse Ackermann bound, range-query structures and the sparse table, persistence, van Emde Boas and word-level parallelism, hashing theory from universal families to cuckoo hashing, and succinct rank and select.
The techniques that show up in ML systems: hashing and the Johnson-Lindenstrauss lemma proved, sketching and streaming, the singular value decomposition and low-rank approximation, spectral graph theory and clustering, linear programming and duality, gradient descent and its convergence rates, sampling and MCMC mixing, and compressed sensing.
Convex sets and functions with the conditions that certify them, Lagrangian duality and KKT worked on examples, gradient descent convergence rates under smoothness and strong convexity, acceleration and why Nesterov's rate is optimal, proximal and stochastic methods, Adam and adaptive preconditioning analyzed, interior-point methods, submodular optimization, and integer programming with relaxation and rounding.
Attacks understood well enough to defend against: buffer overflows and return-oriented programming with the stack drawn out, ASLR/DEP/stack canaries and their bypasses, the browser security model with XSS, CSRF, and injection, authentication and session management, sandboxing and isolation, side channels including Spectre, supply-chain risk, and the security of ML systems including prompt injection and model extraction.
Definitions first: perfect secrecy and its limits, pseudorandom generators and functions, block ciphers and modes with the chosen-plaintext game, message authentication and authenticated encryption done right, hash functions and Merkle trees, number theory for public-key crypto with RSA and Diffie-Hellman derived, elliptic curves, TLS as a case study, zero-knowledge proofs and commitments, and post-quantum lattice cryptography.
Why numerical code fails and how to write code that does not: floating point and catastrophic cancellation with concrete reproductions, conditioning and backward stability, LU, QR, Cholesky, and SVD with when each applies, iterative solvers including conjugate gradient derived, eigenvalue algorithms, automatic differentiation forward and reverse mode implemented, Newton and quasi-Newton methods, and mixed-precision arithmetic on modern hardware.
Making and understanding images: the physics of light transport, the pipeline that renders sixty frames a second, the simulation that moves things believably, and the neural representations that reconstruct a scene from photographs.
The pipeline derived: homogeneous coordinates and the model-view-projection chain multiplied out, rasterization with edge functions and depth buffering, barycentric interpolation and perspective correction, texture mapping with mipmapping and filtering, the physically based shading model and the rendering equation, ray-triangle intersection and acceleration structures, and modern GPU pipeline stages.
Rendering under a 16-millisecond budget: the GPU graphics pipeline stage by stage, deferred and forward-plus shading, shadow mapping and its artifacts, ambient occlusion and global illumination approximations, temporal antialiasing and upsampling, level of detail and culling, compute shaders, and hardware ray tracing with denoising.
Motion that looks right: interpolation and quaternion rotation, skeletal animation and skinning, numerical integration and stability with explicit vs implicit schemes compared, mass-spring and finite element deformation, rigid body dynamics with collision response, cloth and hair, fluid simulation from the Navier-Stokes equations, position-based dynamics, and differentiable simulation for learning.
The rendering equation solved properly: radiometry defined, Monte Carlo integration with importance sampling and its variance analysis, path tracing with Russian roulette, bidirectional and photon-based methods, multiple importance sampling derived, microfacet BSDFs, subsurface scattering and participating media, sampling patterns, and differentiable rendering.
Reconstructing and generating 3D: classical structure from motion and multi-view geometry, implicit surfaces and signed distance fields, NeRF's volume rendering integral derived and discretized, positional encoding and hash grids, 3D Gaussian splatting and its rasterizer, mesh and point cloud networks, generative 3D via score distillation, and the evaluation metrics that mislead.
From the kinematic chain and its control law up to the vision-language-action models that now drive manipulation, including the human-in-the-loop problems that make deployment hard.
The manipulator worked out in full: rigid-body transforms and forward kinematics, the Jacobian and its singularities, inverse kinematics by damped least squares, Lagrangian dynamics for a two-link arm derived by hand, joint-space and operational-space control with stability arguments, impedance and force control, trajectory generation, and the redundancy null space.
Where manipulation gets hard: contact modeling and friction cones, grasp quality metrics and force closure, hybrid force-position control, compliant assembly and the peg-in-hole problem, whole-body and multi-arm coordination, task and motion planning, dexterous in-hand manipulation, and tactile sensing.
Robots that act around people: human motion prediction, game-theoretic interaction models, inverse reinforcement learning and reward inference from demonstrations and preferences, shared autonomy and arbitration, legibility versus predictability, active querying, trust and transparency, and safety guarantees under uncertainty about human intent.
Many robots, one objective: distributed consensus and formation control with their convergence proofs, task allocation via auctions and matching, multi-robot path planning and conflict-based search, coverage and exploration, communication constraints and decentralized estimation, multi-agent RL for teams, and safety in shared human-robot workspaces.
The learning-based robot stack as it stands now: behavior cloning and the covariate-shift problem with DAgger's bound, action chunking, diffusion policies derived for action sequences, vision-language-action models and cross-embodiment training, sim-to-real and domain randomization, world models for planning, large-scale robot datasets, and evaluation protocols that survive real hardware.
Where the models meet a domain and a user: genomics and biomedicine, audio and music, and the design and engineering of the applications people actually touch.
Biology as a sequence-modeling problem: the central dogma stated for engineers, convolutional and transformer models for regulatory genomics, variant effect prediction, protein language models, structure prediction from coevolution through attention-based folding and diffusion-based design, single-cell representation learning, multi-omics integration, causal inference and Mendelian randomization, and interpretability where the science is the point.
The DSP under every audio model: sampling and aliasing, the DFT and FFT derived, windowing and spectral leakage, filter design FIR and IIR with the z-transform, the phase vocoder and time-frequency manipulation, pitch and formant analysis, physical modeling and digital waveguides, reverberation, and the perceptual coding ideas behind lossy audio.
Music as structured signal and symbol: pitch detection and chroma features, beat tracking and tempo estimation, chord recognition and key finding, source separation, symbolic representations and music theory encoded computationally, structural segmentation, similarity and recommendation, and generative music from Markov chains to transformer and diffusion models.
Building an app the current way: SwiftUI's declarative model and its state system (@State, @Observable, bindings), navigation and layout, Swift concurrency with async/await and actors, data persistence with SwiftData, networking and error handling, testing and performance instrumentation, on-device inference with Core ML and the Neural Engine, and the App Store release path.
The design process an engineer can actually run: needfinding and interviews, problem framing and how-might-we, structured ideation, low- to high-fidelity prototyping in Figma with auto layout, components, variants, and prototyping interactions, design systems and tokens, visual and typographic fundamentals, accessibility, usability testing and its sample-size math, and AI-assisted design workflows.
The current stack from first principles: the browser rendering path and the layout/paint/composite pipeline, client vs server vs static rendering and streaming with React Server Components, state management and data fetching, TypeScript at the API boundary, authentication and sessions, edge deployment and caching, Core Web Vitals with the measurements that move them, and accessibility as a correctness property.