These are the thirty papers from the last ten years I would re-implement to actually learn modern machine learning, chosen for three properties at once. Each one changed how systems are built, each one is small enough at reduced scale that a single GPU and real effort can reproduce its core result, and each one rests on a single key idea you should be able to draw on a whiteboard from nothing. Re-implementation is the difference between having read a paper and owning it, because the paper tells you what worked and the re-implementation teaches you everything that silently did not.
Every entry lists what to build at tractable scale, the real dataset to build it on, a difficulty rating from one to five dots, and the key idea behind the method. Links point at the canonical paper and the official code where one exists, each verified. Papers pair naturally with the systems articles here, the recommendation pair grounds video recommendation, the robotics block grounds robot generalization and teleoperation, RAG grounds the RAG design, and FlashAttention grounds LLM serving.
How to use this list. Pick one paper per week or two, build it at the stated scale before reading anyone else's implementation, and only then diff yours against the official code. Keep a notebook of what broke, because the bugs are the curriculum. Difficulty five entries are infrastructure-grade, treat those as reading plus running rather than rebuilding from zero.
Language models and LLM methods
| Paper | Build this | Dataset | Difficulty |
|---|---|---|---|
| Attention Is All You Need (Transformer) (2017) key idea: the stacked encoder-decoder with multi-head attention feeding add-and-norm blocks | A small decoder-only or encoder-decoder transformer written from raw matmuls, trained on character-level text or a small translation pair | Tiny Shakespeare or Multi30k | ●●● |
| BERT (2018) · code key idea: random masking feeding bidirectional attention with the CLS token carrying classification | A tiny BERT pretrained with masked-language modeling, then fine-tuned on sentiment | WikiText-103, then SST-2 | ●●● |
| GPT-2, Language Models are Unsupervised Multitask Learners (2019) · code key idea: the decoder-only stack under a causal mask | The 124M model reproduced end to end, tokenizer to sampling, in the spirit of nanoGPT | OpenWebText | ●●● |
| Scaling Laws for Neural Language Models (2020) key idea: loss versus compute falling on straight lines in log-log space | A grid of small language models across sizes and token budgets, fitting the power laws yourself | WikiText-103 subsets | ●●● |
| LoRA (2021) · code key idea: the frozen weight plus the low-rank BA update in parallel | Rank-r adapter matrices injected into attention weights, compared against full fine-tuning for accuracy and memory | SST-2 or an instruction subset | ●● |
| FlashAttention (2022) · code key idea: query and key tiles streaming through SRAM without materializing the full attention matrix | Tiled attention in Triton with online softmax rescaling, benchmarked against naive attention for memory and speed | synthetic long sequences | ●●●●● |
| InstructGPT (RLHF) (2022) key idea: the three-panel SFT, reward model, PPO pipeline figure | The three-stage pipeline on a small model, supervised fine-tune, reward model on preferences, then PPO against the reward | TL;DR summarization feedback or Anthropic HH | ●●●● |
| Direct Preference Optimization (2023) · code key idea: the preference pair flowing into a single classification-style loss with the reference model | The DPO loss replacing that whole PPO stage, trained straight on preference pairs | Anthropic HH or UltraFeedback | ●● |
| Retrieval-Augmented Generation (2020) key idea: the query hitting the retriever and the generator conditioning on retrieved passages | A dense retriever over a wiki slice feeding a small generator, with retrieval marginalized over the top passages | Natural Questions with a Wikipedia subset | ●●● |
| Mamba (2023) · code key idea: the selective scan replacing attention inside the residual block | The selective state-space block on character-level language modeling, compared against a same-size transformer | Tiny Shakespeare or enwik8 | ●●●● |
Vision
| Paper | Build this | Dataset | Difficulty |
|---|---|---|---|
| ResNet (2015) · code key idea: the residual block, identity shortcut added to the convolution path | ResNet-18 from scratch past 93 percent on CIFAR-10, then the ablation without skips to watch it fail | CIFAR-10 | ● |
| Vision Transformer (ViT) (2020) · code key idea: an image sliced into 16 by 16 patches entering a plain transformer | ViT-Tiny on CIFAR-10, images cut into patches with a learned class token | CIFAR-10 | ●● |
| CLIP (2021) · code key idea: the image-text similarity matrix with matched pairs on the diagonal | A miniature dual encoder trained with the contrastive loss over image-caption pairs, then zero-shot classification from prompts | Flickr30k or COCO Captions | ●●● |
| YOLO (2015) · code key idea: the S by S grid over the image with per-cell box and class predictions | The v1 single-shot detector, a grid of cells each predicting boxes and classes in one forward pass | PASCAL VOC 2007 | ●●● |
| DETR (2020) · code key idea: object queries attending to image features and matching one to one with ground truth | Detection as set prediction, a CNN backbone into a transformer decoded by learned object queries with Hungarian matching | COCO subset | ●●●● |
| Segment Anything (SAM) (2023) · code key idea: heavy image encoder once, light prompt encoder and mask decoder per click | The promptable mask decoder fine-tuned or rebuilt over the released image encoder | a small segmentation set like ADE20K slices | ●●●● |
| Masked Autoencoders (MAE) (2021) · code key idea: the encoder seeing only visible patches and a light decoder rebuilding the rest | ViT pretraining by reconstructing 75 percent masked patches, then linear probing | CIFAR-10 or ImageNet-100 | ●●● |
| DINO, Emerging Properties in Self-Supervised ViTs (2021) · code key idea: student and teacher towers with the EMA update and centering stopping collapse | Student-teacher self-distillation of a small ViT with an EMA teacher, then looking at the attention maps | ImageNet-100 or STL-10 | ●●●● |
Generative, 3D, and speech
| Paper | Build this | Dataset | Difficulty |
|---|---|---|---|
| Denoising Diffusion Probabilistic Models (DDPM) (2020) · code key idea: the forward chain adding noise and the learned reverse chain removing it step by step | A UNet denoiser trained to reverse a fixed noising schedule and sample images from pure noise | MNIST then CIFAR-10 | ●● |
| Latent Diffusion (Stable Diffusion) (2021) · code key idea: pixels compressed by the VAE, the UNet denoising latents, text entering by cross-attention | Diffusion moved into the latent space of a trained autoencoder with cross-attention conditioning | CelebA-HQ small | ●●●● |
| NeRF (2020) · code key idea: rays sampled through space with the MLP queried per point and composited | Tiny NeRF, an MLP mapping position and view direction to color and density, volume rendered per ray | the synthetic Lego scene | ●●● |
| 3D Gaussian Splatting (2023) · code key idea: millions of anisotropic gaussians splatted and alpha-blended per tile | Training the released implementation on your own captured scene, then reading the differentiable rasterizer | Mip-NeRF 360 scenes or your own video | ●●●●● |
| Whisper (2022) · code key idea: log-mel spectrogram into the encoder with task tokens steering the decoder | Fine-tuning whisper-small on accented or domain speech, or rebuilding the encoder-decoder over log-mel inputs | Common Voice | ●●● |
RL and robotics
| Paper | Build this | Dataset | Difficulty |
|---|---|---|---|
| Proximal Policy Optimization (PPO) (2017) · code key idea: the clipped surrogate objective flat outside the trust region | PPO from scratch, advantage estimation and the clipped objective, on classic control then Atari | CartPole, LunarLander, then Pong | ●● |
| AlphaZero (2017) key idea: the search tree guided by the network whose training data is the search itself | Self-play with MCTS and a policy-value network on a small board game, the same loop as my chess project | Connect Four or Othello self-play | ●●●● |
| Diffusion Policy (2023) · code key idea: noisy action trajectories denoised into a coherent motion conditioned on the scene | A diffusion model over action sequences conditioned on observations for a pushing task | Push-T benchmark | ●●●● |
| ACT / ALOHA, Learning Fine-Grained Bimanual Manipulation (2023) · code key idea: the CVAE encoder and the transformer emitting chunks of future actions | Action chunking with transformers trained on scripted or teleoperated episodes in the ALOHA sim | ALOHA simulated transfer-cube | ●●●● |
| OpenVLA (2024) · code key idea: camera frames through vision encoders into the language model emitting action tokens | LoRA fine-tuning of the released 7B model on one robot task family rather than pretraining | LIBERO task suite | ●●●●● |
Recommendations
| Paper | Build this | Dataset | Difficulty |
|---|---|---|---|
| Deep Neural Networks for YouTube Recommendations (2016) key idea: the funnel from millions of items through candidate generation into ranking | The two-tower candidate generator, users and items embedded and matched by nearest neighbor | MovieLens-25M | ●● |
| DLRM, Deep Learning Recommendation Model (2019) · code key idea: sparse embeddings and dense features meeting in the interaction layer | Meta's click-through architecture, embedding tables for sparse features crossed with dense features | Criteo Kaggle CTR | ●●● |