Karpathy Autoresearch Skill for Claude Code & Codex

karpathy-autoresearch

Turn one measurable optimization problem into a controlled propose, evaluate, keep-or-revert cycle.

Establish the experiment contract

Require these inputs before changing anything:

• the repository and reproducible environment;
• one primary metric and whether higher or lower is better;
• the exact evaluation command;
• a fixed wall-clock, compute, or sample budget per experiment;
• files the agent may edit and files it must not edit;
• guardrail checks that can reject a result even when the metric improves;
• maximum iterations, total budget, and other stopping conditions.

Run the untouched baseline at least once. Record the revision, environment, command, budget, primary score, guardrail results, and meaningful variability. If the baseline cannot be reproduced well enough to compare changes, stop and repair the experiment before optimizing it.

Freeze what makes results comparable

Do not change the evaluator, evaluation data, primary metric, comparison direction, or per-experiment budget during a run. Treat environment changes as a new run with a new baseline.

Allow edits only inside the declared surface. Andrej Karpathy's source setup protects data preparation and evaluation utilities while allowing the agent to edit one training file. This skill generalizes that boundary to other measurable code experiments; that generalization is an XSkills adaptation.

Run one experiment

For each iteration:

1. Read the accepted revision and experiment ledger.
2. State one hypothesis that has not already been tested.
3. Predict why the change should affect the primary metric.
4. Make the smallest coherent change needed to test it.
5. Run the unchanged evaluation under the fixed budget.
6. Run every declared guardrail check.
7. Compare the result with the current accepted score.
8. Keep the change only when the primary metric improves and every guardrail passes.
9. Otherwise restore the accepted revision completely.
10. Append the result to the ledger before choosing another hypothesis.

Do not stack an unmeasured second idea onto the same iteration. If an experiment crashes, record the failure and revert it; do not silently extend its budget or alter the evaluator.

Maintain the ledger

Use one row per experiment:

Include failed and rejected experiments. The negative results prevent future agents from repeating the same idea and make the final result auditable.

Stop safely

Stop when any declared condition is met, including the iteration limit, total budget, repeated infrastructure failures, convergence threshold, or lack of non-duplicate hypotheses. Return:

• the original baseline and best accepted score;
• the exact accepted revision;
• guardrail evidence for that revision;
• the complete experiment ledger;
• the stopping reason;
• promising hypotheses that were not tested.

Never claim that an improvement transfers to different hardware, data, workloads, or environments without a separate evaluation.

Source attribution

Independently compiled by XSkills from Andrej Karpathy's autoresearch repository. Karpathy did not author or review this skill. The source applies a fixed five-minute budget, one editable training file, one validation metric, and a keep-or-discard loop to autonomous model-training experiments. Generalizing that contract to other measurable code optimization tasks, adding guardrail checks, and formalizing stopping conditions are XSkills adaptations. Attribution does not imply endorsement.