autoresearch

Part I: The mental model

program.md            human-authored context: the goal, the rules, the loop, "NEVER STOP"
      |
      |  a coding agent reads this, not a Python interpreter
      v
coding agent          Claude Code / Codex / Cursor, kicked off by one prompt
      |
      v  forms a hypothesis from the ML literature
edit train.py         the ONLY editable file: model, optimizer, loop, hyperparameters
      |
      v
git commit            snapshot the candidate on a branch
      |
      v
uv run train.py       train for exactly TIME_BUDGET = 300s (5 min) of wall clock
      |
      v
val_bpb               fixed metric from prepare.py, bits per byte, lower is better
      |
      +---- improved? ----- yes --> keep the commit, append a row to the results log
      |                      no  --> git reset, throw the change away
      v
loop                  ~12 experiments/hour, an overnight run is ~100, until you stop it

The one-sentence identity. autoresearch is the claim, made as small and concrete as possible, that the unit of ML-research automation can be a single markdown file of research-org policy driving a general coding agent over a deliberately tiny training run, so the human programs the context and the agent programs the code. That inversion is the whole point. In an ordinary project you write Python and a machine runs it. Here you write program.md, an English description of what to optimize and how to behave, and a language model runs that, treating your train.py as the raw material it is free to rewrite. Karpathy's framing is that you are programming the program.md, and the Python underneath has become the substrate the agent edits rather than the thing you author by hand.

Two consequences follow, and both are more interesting than the code. First, the leverage moves. Your taste, your priors about what is worth trying, and your rules about what counts as progress all live in a prose file, and the actual code changes are delegated. Second, the system only works because everything else is nailed down. The metric is fixed, the time budget is fixed, the dataset is fixed, and exactly one file is editable, so the agent has a narrow, honest arena in which a change either lowers one number or it does not. Remove any of those constraints and the loop stops being trustworthy. autoresearch is small on purpose. It is a demonstration, not a platform, and reading it is worth an hour precisely because the idea is bigger than the repository.

Part II: Using it

autoresearch targets a single NVIDIA GPU, was developed on an H100, and uses uv as its package manager. The one-time setup downloads data and trains a tokenizer, then a single experiment is one command.

curl -LsSf https://astral.sh/uv/install.sh | sh   # install uv, once
git clone https://github.com/karpathy/autoresearch
cd autoresearch
uv sync                 # resolve the (small) dependency set, essentially torch plus a few packages
uv run prepare.py       # download data shards and train the ~8k-token BPE tokenizer, once
uv run train.py         # run one baseline experiment, about 5 minutes on one H100

That last command is the atom of the whole system. It trains a small GPT for a fixed five minutes and prints a short block of summary metrics on the way out. The two lines that matter are the score and the memory, printed in a form that is trivial to machine-read.

uv run train.py > run.log 2>&1
grep "^val_bpb:\|^peak_vram_mb:" run.log
# val_bpb: 0.9979
# peak_vram_mb: 45123

val_bpb is validation bits per byte, and lower is better. The baseline sits a touch under 1.0, and the baseline value plus a peak-VRAM figure of roughly 45 GB are written into program.md as the numbers a candidate must beat. Now the part that surprises people. There is no orchestration script. You do not run autoresearch --loop. The automation layer is the language model itself. You open a coding agent in the project directory and prompt it with something close to the phrasing Karpathy suggests.

# in the repo directory, inside Claude Code, Codex, Cursor, or similar:
# "Have a look at program.md and let's kick off a new experiment."

From there the agent reads program.md, which tells it to form a hypothesis, edit train.py, commit, run the five-minute experiment, read val_bpb, keep or revert, and repeat without pausing to ask you. A representative edit is small. The agent might raise the Muon matrix learning rate by a hair, swap an activation, add a scalar it thinks is missing, or delete code it judges to be dead weight.

# train.py is the agent's arena. A minimal, representative hypothesis:
# "the matrix learning rate looks conservative, try nudging it up"

matrix_lr = 0.04      # baseline
matrix_lr = 0.05      # candidate the agent commits, runs, and then keeps or reverts

Now the mistakes to expect. First, do not try to make experiments fair by giving them more time. The budget is fixed at 300 seconds in prepare.py on purpose, so a change that helps has to help within five minutes of wall clock, and lengthening the run silently changes the game. Second, do not edit prepare.py or the metric. The evaluation is marked as fixed and read-only, and an agent that "improves" val_bpb by touching how it is computed has cheated rather than discovered anything. Third, expect crashes. Agents propose broken code often, and the intended behavior is to fix trivial mistakes like a typo or a missing import and to throw away any idea that is fundamentally broken rather than nurse it. Fourth, and most important, your real work is program.md, not babysitting the runs. The quality of an overnight session is mostly decided by the quality of the prose you wrote before you went to bed.

Part III: When it is the right tool

autoresearch is the right thing to reach for when you want to feel agentic ML research end to end on hardware you actually have, when you are teaching or learning what an autonomous research loop is and want every moving part visible in three files, or when you have a genuinely small, well-scoped optimization target and a metric you trust enough to hand to a greedy hill climber overnight. It is a wonderful demonstration and a wonderful teaching object. It is emphatically not a general research platform, and Karpathy presents it as an experiment rather than a product.

The honest neighbors. The training code descends from the nanoGPT and modded-nanoGPT speedrun lineage, and if your goal is to understand or beat a single-GPU pretraining speedrun by hand, that community and its leaderboard are the real center of gravity, with autoresearch sitting on top as the agent that plays the same game. If you want an autonomous system that writes whole papers, with idea generation, experiment code, figures, and a written manuscript, Sakana's AI Scientist aims at that much larger and much more speculative target. If you want serious multi-GPU pretraining with real parallelism rather than a five-minute toy, torchtitan is the reference platform, and the entire world of device meshes and collectives it lives in is the subject of my parallel computing notes. autoresearch deliberately renounces all of that, single GPU, single file, no distributed anything, because the point is to isolate the agentic loop, not to chase throughput.

The comparison people reach for first is classic hyperparameter search, and the difference is the load-bearing one. A grid or Bayesian sweep explores a fixed, pre-declared space of numbers you chose in advance. autoresearch's agent edits source code, so it can change the optimizer, restructure attention, add or remove a mechanism, or delete a block entirely, which is a strictly larger and messier space than any sweep can name ahead of time.

hyperparameter sweep         agent editing code (autoresearch)
--------------------         -----------------------------
you enumerate the axes       the agent invents the axes as it goes
values only                  values, structure, and whole mechanisms
never crashes on syntax      crashes constantly, must self-repair
reproducible, boring         open-ended, and mostly finds small wins

The architecture-shaped warning is about the ratchet. Because a change is kept only if it immediately lowers val_bpb, the search is greedy, and greedy search cannot take a step backward to set up a larger gain later. That is a real ceiling on creativity, examined in Part V, and it is why the honest read of autoresearch is "a diligent tireless tweaker" rather than "a machine that invents new architectures". Knowing which of those two you need decides whether this is your tool.

Part IV: The full life of one experiment

The specimen is one full turn of the loop, from the moment the agent decides what to try to the moment the change is kept or thrown away. Everything below happens inside the coding agent you launched, driven by the rules in program.md.

Stage 1: read the context and the current state

The agent begins by reading program.md for its standing orders, then reads the current train.py to see what the code looks like right now, and reads the running results log to see what has already been tried and what worked. This last step matters more than it looks. The log is the agent's memory across experiments, and a good program.md tells it to consult that memory so it does not keep re-proposing the same dead ideas. The state going into every turn is therefore the three things a human researcher would also hold in mind, the goal, the current code, and the history of what has been tried.

Stage 2: form a hypothesis

The agent proposes one concrete, testable change and, ideally, says why. This is where the language model's compressed knowledge of the ML literature does the work, since it can reach for ideas like a different learning-rate schedule, a normalization tweak, a change to the attention window pattern, or a small architectural substitution. The discipline that program.md imposes is one idea at a time. If you change five things and the number moves, you have learned nothing about which of the five mattered, so the loop is built around isolated, attributable changes.

Stage 3: edit train.py and commit

The agent edits train.py, the single editable file, and commits the candidate to a git branch so there is a clean snapshot to keep or to reset. Git is doing real work here as the keep-or-revert mechanism, not just as version history. Everything in train.py is fair game, the model architecture, the optimizer, the hyperparameters, the training loop, the batch size, even the model size, as long as the result still trains and still prints a val_bpb.

Stage 4: run the five-minute experiment

The agent runs the training command and captures the output, in the shape uv run train.py > run.log 2>&1. The script trains for exactly TIME_BUDGET seconds of wall clock, which is 300 by default, and this fixed budget is the great equalizer of the whole system. Every experiment gets the same five minutes, so a change is judged on what it achieves per unit of wall-clock training time, which is exactly the currency a real speedrun cares about. A change that would help given infinite steps but is slower per step is correctly counted as a loss here, and that is a feature.

Stage 5: crash handling

Agent-written code fails often, and the loop expects it. The rule in program.md is to repair obviously trivial breakage, a typo, a missing import, a shape mismatch the agent can see, and to discard anything fundamentally broken rather than spend the night debugging one bad idea. There is also a wall-clock guard around the run, so a hang does not consume the whole session, it is killed and the experiment is written off. The emotional tenor of this stage is important. Failures are cheap and expected, and the correct response to most of them is to move on.

Stage 6: read the metric and decide

When the run finishes, the agent greps the log for the summary lines, reads val_bpb, and compares it to the best so far. The decision is a pure ratchet. If the number improved, the commit stays and a row is appended to the results log recording the idea and its metrics. If the number is equal or worse, the agent runs git reset to throw the change away and returns the code to the last good state. Peak VRAM is tracked alongside, treated as a soft constraint, so a change that costs memory is tolerated when the gain justifies it and rejected when it does not. There is also a tie-breaker worth internalizing. When two variants score about the same, simpler wins, and a change that deletes code while holding the metric flat is a good change.

Stage 7: never stop

The last instruction is the one that makes it a loop rather than a single suggestion. program.md tells the agent, in effect, that once the experiment loop has begun it must not pause to ask the human whether to continue, and should keep going until it is manually stopped. That single directive is what turns a helpful assistant into an overnight research process. You write the rules, you kick it off, and you read the log in the morning. That closes the life of one experiment, and the same seven stages simply repeat, roughly a dozen times an hour, on their own.

Part V: Internals deep dives

Deep dive: program.md is the program

The most important file in autoresearch contains no code. On current main program.md reads like the operating manual of a one-person research org, and its sections cover the setup, what may and may not be changed, the exact output format the metric is printed in, how results are logged, and the experiment loop itself. It hardcodes the baseline numbers a candidate must beat, states the fixed five-minute budget, marks the memory as a soft constraint, gives the precise command to run an experiment, prescribes the keep-on-improve and revert-on-fail behavior, and lays down the "NEVER STOP" directive.

Read that list again and notice what it is. It is policy, written in English, for an agent whose mechanism is a general coding model. This is the sense in which Karpathy says you program the program.md. The interesting engineering surface is no longer the training loop, it is the prose that constrains an open-ended optimizer just enough to be useful, and that framing lines up with the broader move toward treating an agent's context as the real artifact you author. Two lessons fall out immediately. The more precisely program.md pins down the metric, the budget, the read-only surface, and the accept criterion, the more trustworthy the overnight run, because every ambiguity is a place the agent can drift or accidentally cheat. And the ideas that get tried are bounded by what the prose invites, so a program.md that encodes real research taste produces a better night than a vague one. Karpathy's own remark is the honest version of this. Writing a good program.md requires having done the research yourself, which is why this is a lever for experienced researchers rather than a replacement for them.

Deep dive: the three-file contract

The whole design rests on a strict separation of roles across three files, and the discipline of that split is most of what makes the loop safe.

prepare.py is the referee, and it is deliberately off limits. It downloads the data shards, trains the byte-pair tokenizer with a small vocabulary of roughly eight thousand tokens, holds the fixed constants like the 2048-token context length and the 300-second TIME_BUDGET, and defines the evaluation, with the metric explicitly marked as not to be changed. Keeping the referee in a file the agent may not touch is what stops the classic failure of a self-improving system, which is improving the scoreboard instead of the play. train.py is everything the agent is allowed to reshape, a few hundred lines holding a complete small GPT, its optimizer, and its loop. The boundary between the two is the boundary between what is being optimized and what is doing the judging, and drawing that boundary as a plain filesystem-and-prose rule, rather than as some sandbox, is a large part of the charm.

Deep dive: train.py, the model and MuonAdamW

The editable file is a compact, modern, single-GPU GPT in the modded-nanoGPT speedrun style, and reading it is a good tour of the tricks that lineage has accumulated. The model is a decoder- only transformer with rotary position embeddings, and the block uses a squared-ReLU MLP rather than GELU, QK-normalization inside attention, and a sliding-window attention pattern that alternates short and long windows across layers. There are learnable value embeddings mixed in on some layers. The defaults are small, on the order of a depth near eight and a width near 768 with a handful of heads, tuned so that a meaningful amount of training fits inside five minutes. Every one of these is a default the agent is free to change, and several of the improvements found in practice were adjustments to exactly these mechanisms. If any of that vocabulary is unfamiliar, my language models from scratch notes build the same pieces up by hand.

The optimizer is the other half worth understanding, a combined strategy that applies Muon to the hidden weight matrices and AdamW to everything else. Muon is the newer optimizer that orthogonalizes each matrix update, and it has become the engine of the single-GPU speedrun world because it converges faster per step on exactly these hidden linear layers. AdamW handles the parts that are not two-dimensional hidden matrices, the embeddings, the unembedding, the scalar gains, and the value embeddings, each with its own learning rate. The defaults live in train.py as ordinary constants, a large embedding learning rate, a small unembedding learning rate, a moderate Muon matrix rate, a bit of weight decay, and Adam betas skewed toward a fast-moving first moment. These specific numbers are not sacred. They are the very things the agent nudges, and the loop exists to discover which nudges survive the five-minute test. The exact spellings and values here move with the upstream speedrun, so learn the shape, Muon on the matrices and AdamW on the rest, rather than memorizing a constant.

The loop itself is plain single-GPU PyTorch. It prefetches batches, accumulates gradients to reach a target tokens-per-step, updates learning rates on a schedule keyed to a progress fraction computed from elapsed training time rather than from a fixed step count, steps the optimizer, and watches the loss for a non-finite blow-up that aborts the run. The stopping condition is the interesting bit, since the loop breaks once accumulated training time passes TIME_BUDGET, which is why the budget is measured in seconds and not in steps. At the end it calls the fixed evaluation and prints the summary block.

Deep dive: val_bpb, and why it is fixed

The single number the whole system optimizes is validation bits per byte, computed by the evaluation in prepare.py over a held-out validation shard. Bits per byte is cross-entropy measured against the raw bytes of the text rather than against tokens, so the loss in nats per token is summed and then divided by the number of underlying bytes and converted to base two. The reason for that mouthful is the property it buys. Because the metric is normalized by bytes rather than by tokens, it is comparable across changes to the tokenizer or the vocabulary, so an architectural change and a tokenization change can be scored on the same axis without one silently gaming the other. Token- level perplexity would not have that property, and an agent optimizing it could win by changing what a token is.

The metric is fixed for a deeper reason than fairness. An autonomous optimizer will optimize whatever it is graded on, including the grader, so the moment the scoreboard is editable the results become meaningless. autoresearch handles this in the bluntest possible way, by putting the metric in a file the agent is told never to touch and labeling it as the fixed target. That single rule is the difference between a system that discovers faster training and a system that discovers how to print a smaller number. It is the same instinct behind never letting a benchmark live in the same trust boundary as the thing it benchmarks.

Deep dive: the budget and the ratchet

Two design choices give the loop its character. The fixed five-minute budget makes experiments cheap and comparable, roughly a dozen per hour, which is what makes an unattended overnight run of around a hundred experiments sensible in the first place. It also defines what "better" means, since holding wall clock constant turns the search into a search for efficiency, better loss for the same time, which is precisely the speedrun objective.

The git ratchet, keep on strict improvement and revert otherwise, is what makes progress monotone. Every accepted change genuinely lowered the metric, so the code only ever gets better and the run can compound small wins into a stacked result. That is the strength. The matching weakness is unavoidable and worth stating plainly. A strict ratchet is greedy, and greedy search cannot accept a temporary regression to reach a better basin, so autoresearch will happily find a stack of small local wins and will almost never find a change that needs to get worse before it gets better. This is the creativity ceiling. In Karpathy's own two-day run the system behaved exactly like this, a diligent tweaker that improved things it could reach by hill climbing and did not invent a new architecture. If you wanted to lift the ceiling you would have to loosen the ratchet, accept regressions sometimes, run several lineages in parallel, and select later, which is the direction more elaborate evolutionary agent systems take. autoresearch stays simple on purpose, and the ceiling is the price of the simplicity.

One concrete finding from Karpathy's run makes the whole thing feel real. The agent noticed that the QK-normalization was missing a scalar multiplier, which had left attention too diffuse across heads, a small correctness detail on already-tuned code that a human had not caught. That is the flavor of win this system produces, not a revolution, but exactly the kind of careful, literature-informed fix a tireless junior researcher might turn up on the hundredth try.

Part VI: Reading the repository

The repository is tiny, which is half its value. You can read all of it in an evening, and the right order is not the order of file size. Details are current as of the March 2026 release and this is a fast-moving experiment, so treat exact numbers as a snapshot.

Stage 0, the contract. Read the README.md, then read program.md in full. Do not skim the markdown, it is the actual program. Questions to hold: what number is being optimized, what is the human forbidden to leave ambiguous, and what single sentence turns the assistant into a loop.

Stage 1, the referee. Read prepare.py. Find the fixed constants, the context length and the TIME_BUDGET, find where the tokenizer is trained and how big its vocabulary is, and read the evaluation function that computes val_bpb. Questions: why is bits per byte the metric rather than token perplexity, and why does this file, not train.py, own the metric?

Stage 2, the arena. Read train.py top to bottom. Meet the GPT and its modded-nanoGPT-style tricks, the rotary embeddings, the squared-ReLU MLP, QK-norm, the sliding window pattern, and the value embeddings. Then meet the combined Muon-plus-AdamW optimizer and see which parameters go to which. Then read the training loop and find the exact line where the five-minute budget ends the run. Questions: what makes this a single-GPU file with no distributed anything, and which constants are the ones an agent would most naturally tweak first?

Stage 3, one real experiment. Run uv run train.py yourself and read run.log. Match the summary block to the metric definition you read in stage 1, and confirm the two grep-able lines are exactly what the loop keys its decisions on.

Stage 4, the loop and its trail. Now drive a coding agent through a handful of experiments and watch the git history and the results log fill in. The analysis notebook shipped with the repo plots the trajectory of the metric over experiments, which is the clearest picture of the ratchet at work, a staircase that only ever steps down. Questions: how does the agent use the results log as memory, and where in the history can you see a reverted idea leave no trace in the code but a row in the log?

Where not to start. Do not begin by tuning hyperparameters yourself, and do not begin by reading community forks that add macOS, Windows, or AMD support, since they change the substrate and blur the idea. The lesson lives in the three-file contract and the prose, and the fastest way to miss it is to treat autoresearch as a training script rather than as a claim about where research automation is heading.

Part VII: Hands-on labs

Labs 1 and 2 need one GPU and a few minutes each. Labs 3 and 4 need a coding agent. Lab 5 needs neither and is the one that teaches the most.

Lab 1: run the baseline. Concept: the atom of the system.

uv sync
uv run prepare.py            # one time
uv run train.py > run.log 2>&1
grep "^val_bpb:\|^peak_vram_mb:" run.log

Read the whole summary block, then find the exact val_bpb and compare it to the baseline recorded in program.md. Confirm for yourself that the run really did stop on wall clock and not on a step count, by looking for the time-budget check in train.py.

Lab 2: be the agent, by hand. Concept: the ratchet, run manually.

git checkout -b hand-experiment
# edit ONE thing in train.py, for example nudge the Muon matrix LR from 0.04 to 0.05
uv run train.py > run.log 2>&1
grep "^val_bpb:" run.log
# improved?  git commit -am "raise matrix lr"      keep it
# worse?     git checkout -- train.py              throw it away

Doing one turn by hand makes the loop concrete. You will feel how cheap a five-minute experiment is, how binary the keep-or-revert decision is, and how quickly you run out of confident hypotheses, which is exactly the point where handing the wheel to an agent starts to pay off.

Lab 3: kick off the real loop. Concept: the LLM as the automation layer.

# open a coding agent in the repo directory, then prompt:
# "Read program.md, then run the experiment loop. Try 5 experiments, one idea each,
#  keep improvements and revert the rest, and log every result."

Watch the agent form a hypothesis, edit train.py, commit, run, and decide, five times. Then read the git log and the results file together. The lesson is that nothing in this loop is a script you wrote, the control flow lives in the prose of program.md and the judgment lives in the model.

Lab 4: change the research direction from prose. Concept: programming the program.md.

# edit program.md, not the code. For example add a constraint:
#   "Prefer changes that also reduce peak VRAM. Reject any change that raises it."
# then kick off another short session and see how the agent's choices shift

You changed no Python and yet the behavior of the whole research process changed. That is the entire thesis of the repository, experienced in one edit. Try a second variation that tells the agent to favor deleting code, and watch simplicity pressure show up in what it proposes.

Lab 5: reason about the ceiling. Concept: greedy search, no GPU required.

On paper, design a change that would lower val_bpb in the long run but raise it within five minutes, for example a warmup-heavy schedule that pays off late, or a larger model that is slower per step. Predict what the strict ratchet does to it, then predict how you would have to change program.md and the accept rule to let such an idea survive. This lab is where the difference between a tweaker and an inventor becomes something you can state precisely, and it is the most valuable half hour in this chapter.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is autoresearch, in one sentence?

A small experiment in which a general coding agent, driven by a markdown policy file, runs its own overnight ML-research loop on a single-file, single-GPU training script, keeping code changes that lower one fixed metric and reverting the rest.

2. What does it mean that you program the program.md?

The human's authored artifact is prose, not code. You write the goal, the rules, the accept criterion, and the never-stop directive into program.md, and a language model executes that policy by editing the Python for you, so your leverage is in the context you give the agent rather than in the training code itself.

3. Why is exactly one file editable?

To keep the arena small and the decisions attributable. train.py holds the model, optimizer, and loop, so every hypothesis is a change to it, while the data, tokenizer, and metric live in a read-only file so the agent cannot move the goal posts. One editable file also keeps diffs small and reviewable.

4. Why is the metric fixed and kept out of the agent's reach?

An autonomous optimizer will optimize whatever it is graded on, including the grader. Putting val_bpb in prepare.py and forbidding edits to it is what keeps the system discovering faster training rather than discovering how to print a smaller number.

5. Why bits per byte instead of token perplexity?

Bits per byte normalizes the loss by the underlying bytes of text rather than by tokens, so it is comparable across tokenizer or vocabulary changes. A metric normalized by tokens could be gamed by redefining what a token is, which is exactly the kind of edit an agent is capable of making.

6. Why is the budget five minutes of wall clock rather than a step count?

Because the objective is efficiency, better loss per unit of real training time, which is the speedrun currency. Fixing wall clock makes every experiment comparable and correctly counts a change that is slower per step as a loss even if it would help given infinite steps.

7. What is the keep-or-revert rule, and what is its cost?

Keep the commit if val_bpb strictly improved, otherwise git reset and discard. That makes progress monotone and lets small wins stack, but it is greedy, so it cannot accept a temporary regression to reach a better solution, which caps the system at hill-climbing rather than invention.

8. How is the agent actually run? Is there an orchestrator?

There is no orchestration script. You open a coding agent such as Claude Code, Codex, or Cursor in the project directory and prompt it to read program.md and begin. The language model is the automation layer, and the never-stop directive in the prose is what keeps it looping.

9. Why does prepare.py sit off limits while train.py is fully open?

Because prepare.py is the referee, holding the data, the tokenizer, the fixed constants, and the metric, while train.py is the thing being optimized. Separating the judge from the contestant along a plain filesystem boundary is what keeps the results meaningful.

10. What kind of results did Karpathy report, and how should you read them?

Over roughly two days on a small model the system ran on the order of several hundred experiments and found around twenty genuine improvements, which stacked into an eleven percent reduction in time-to-target on code he had already optimized, including spotting a missing scalar in the QK-norm. Read that as evidence of a diligent tweaker producing real but incremental wins, not as an architecture inventor.

11. When would you not use autoresearch?

When you need real multi-GPU pretraining, reach for torchtitan. When you need a full paper-writing pipeline, that is the target of larger systems like the AI Scientist. When you need genuine architectural novelty rather than local tuning, the greedy ratchet is the wrong engine. autoresearch is a demonstration and a teaching tool, not a production research platform.

12. What is the single biggest risk in an unattended run, and how does the design mitigate it?

Reward hacking, the agent improving the score without improving the model, usually by touching how the score is computed. The design mitigates it structurally by making the metric and its data read-only in a separate file and by keeping the whole editable surface down to one training script that must still produce an honest val_bpb.

Part IX: Design lessons

Program the context, not the code. The authored artifact here is program.md, a policy in prose, and the code is what the agent writes. Wherever an agent is the executor, the leverage moves into the context you hand it, and treating that context as a first-class program is the shift worth internalizing. It is the same instinct as writing a good spec or a good prompt, taken seriously enough to run overnight.

Fix the metric and the budget, let everything else move. Autonomy is only safe when the grader and the clock are outside the agent's reach. autoresearch pins val_bpb and the five-minute budget in a read-only file and opens up literally everything else, which is the minimal structure that makes an open-ended optimizer trustworthy. Any self-improving system needs this line drawn somewhere.

Separate the referee from the contestant. The three-file contract keeps the thing being optimized and the thing doing the judging in different files with different edit rights. That boundary is what stops the oldest failure of self-improvement, optimizing the scoreboard, and it costs nothing but discipline.

Make experiments cheap enough to be disposable. When one experiment is five minutes and a git reset, failure is not expensive and the agent can afford to be wrong most of the time. Cheap, reversible trials are what make an autonomous loop productive rather than terrifying, and the same pattern shows up in fast test suites and in throwaway feature branches.

A greedy ratchet is a feature and a ceiling. Keep-on-improve gives you monotone, compounding progress and a clean staircase of wins, and it also forecloses any gain that requires a step backward. Knowing that trade is knowing exactly what class of discovery this design can and cannot produce, which is more useful than either hype or dismissal.

Scope down until the idea is visible. Single GPU, single file, one metric, three files total. The renunciation of scale is what makes the agentic loop legible, and there is a general lesson there about demonstrating a new idea at the smallest size that still contains it. If tomorrow's version of this runs as swarms of agents across clusters, the serving and inference substrate for that, the world of systems like vLLM, becomes the bottleneck, but the conceptual core is exactly the small loop shown here.

Part X: Memorization framework

The one-sentence summary. autoresearch fixes a metric, a budget, and a dataset in a read-only file, hands a coding agent a single editable training script and a markdown policy that says hypothesize, edit, run five minutes, keep or revert, and never stop, and lets the loop hill-climb one number overnight.

program.md (policy, human)  ->  coding agent  ->  edit train.py  ->  git commit
  ->  uv run train.py (300s)  ->  val_bpb  ->  improved? keep : git reset  ->  loop

The roles mapped to files:

policy / rules / loop   program.md      (human authors, agent obeys)
fixed substrate         prepare.py      (data, ~8k BPE tokenizer, metric, constants)
the arena               train.py        (GPT + Muon/AdamW + loop, agent edits)
one experiment          uv run train.py -> run.log -> val_bpb, peak_vram_mb
the trail               git history + results log + analysis notebook

Memorize these blocks:

• The inversion: you program the program.md, the agent programs the Python.
• The contract: program.md is policy, prepare.py is the read-only referee, train.py is the only editable file.
• The metric: val_bpb, bits per byte, lower is better, byte-normalized so tokenizer changes cannot game it, and never editable.
• The budget: TIME_BUDGET is 300 seconds of wall clock, which makes the objective efficiency and makes experiments cheap and comparable.
• The ratchet: keep on strict improvement, git reset otherwise, monotone and compounding but greedy, which is the creativity ceiling.
• The result to remember: Karpathy's run found incremental wins on already-tuned code, a diligent tweaker, not an architecture inventor.

Part XI: Papers and further reading

The repository is small but its lineage is real, and each of these works rewards a direct read. Where this site covers the same ground in depth, the companion link points there.

1. Karpathy, nanoGPT, 2022. The minimal GPT training repository this whole lineage descends from, walked through in the nanoGPT chapter on this site.
2. Jordan et al., modded-nanogpt, 2024. The NanoGPT speedrun whose accumulated tricks, squared-ReLU MLPs, QK-norm, sliding-window attention, and Muon, fill the train.py the agent edits.
3. Jordan et al., Muon, An optimizer for hidden layers in neural networks, 2024. The writeup of the orthogonalized-update optimizer that autoresearch applies to the hidden weight matrices.
4. Lu et al., The AI Scientist, Towards Fully Automated Open-Ended Scientific Discovery, 2024. The maximalist end of agentic research, full papers rather than one metric, against which autoresearch's deliberate smallness is easiest to see. The broader question is surveyed in the self-improving agents class.
5. Su et al., RoFormer, Enhanced Transformer with Rotary Position Embedding, 2021. The rotary position encoding the model uses, built by hand in the language models from scratch notes.
6. Henry et al., Query-Key Normalization for Transformers, 2020. The QK-normalization idea where the agent's best-known find, a missing scalar multiplier, lived.
7. Loshchilov and Hutter, Decoupled Weight Decay Regularization, 2017. AdamW, the half of the combined optimizer that handles everything Muon does not.

Part XII: Final takeaway

If the training code underneath feels unfamiliar, the nanoGPT chapter and my language models from scratch notes build the same small GPT and its speedrun tricks from the ground up, and the broader question of how far an autonomous agent can push its own work is the subject of the self-improving agents notes. Then come back and reread program.md once more. It will read like the job description of a research org, and that is the whole point.