Five hypotheses · seven experiments · all code & data public
What reinforcement learning looks like when you raise it the way children are raised — with maps, drills, mess, teachers, and a body that starts small.
Reinforcement learning is usually told as a story about reward. An agent acts, the world scores it, and a policy slowly bends toward whatever pays. Imitation learning is the opposite story: copy the teacher and hope the world never drifts from the demonstrations.
Humans live between those stories. We take instruction — compressed experience handed down by people who practiced before us — and we cash it out through our own trial and error. We keep a model of the world in our heads and rehearse inside it. We drill small pieces before playing whole games. We practice with deliberate variation, because smooth repetition is a comfortable lie. And we do all of it inside a body that starts small and grows on a schedule that seems suspiciously convenient for learning.
This page reports five hypotheses about that middle ground, each turned into a falsifiable experiment on deliberately small systems — tabular and linear agents, gridworlds and a torque-limited pendulum — where every step is countable, budgets are matched to the step, and 20–40 seeds per condition make the statistics honest. Small worlds, so the mechanisms have nowhere to hide.
Protocol, in one line: IQM point estimates, 95% percentile-bootstrap CIs, Mann-Whitney and Welch tests Holm-corrected, confirmatory runs on fresh seeds disjoint from all tuning, and identical evaluation for every condition. Full details in the methods section — including the claims that failed.
Every experiment here is tabular Q-learning: a table with one row per situation, one column per action, holding a running guess of long-run reward.
Reach the goal once and its value leaks one step back along the path. Arrows grow as the table converges. That's the whole engine.
Because the hypotheses are about structure around the learner — models, drills, teachers, bodies. A small learner keeps the structure visible.
H1 · world models — chapter one
Close your eyes in your own home and you can still find the fridge. Close your eyes in a stranger's home and you are lost immediately. Your motor neurons didn't change between those two sentences — your map did.
We raised an agent in a 13×11 apartment with 10% motor slip, letting it learn both a policy and a transition model of its home. Then we took its observations away entirely. Blindfolded, it navigates by dead reckoning: it imagines where each action should have taken it, tracks a belief over where it probably is, and asks its value table what to do from there. The test: same blindfold, familiar home versus stranger's home. Thirty fresh seeds, every comparison Holm-corrected, two test statistics.
With eyes open, the trained agent reaches the fridge in 97% of episodes. The spotlight is its observation.
Pure dead reckoning drops to 20% — slip noise compounds and the belief drifts. That prediction failed, honestly. But add touch — a bump signal filtering the belief, a hand trailing the wall — and success is 97.5%: statistically indistinguishable from sighted.
Same agent, same blindfold, same touch — new floor plan. Success collapses to 29%, against a measured 3% random floor (Welch p=7e-18). The ghost marches confidently through walls that are not where it believes.
Sighted transfer to home B with home A's values also fails (0%). It was never about vision. It's the map — and a hand on the wall to keep the map honest.
Greedy success at the fridge, every 1k steps · fresh seeds 100–129 · IQM with 95% bootstrap CI
Dyna-Q reaches 90% in 2,562 steps vs 25,500 for plain Q-learning — but an update-matched replay baseline (20 replayed updates per real step, no model at all) lands at 2,500, closing the entire gap (p=0.55). The famous speedup is an update-count effect, exactly as van Hasselt (2019) warned. What replay cannot do is walk blind: only the model supports dead reckoning. The world model's value here isn't faster learning — it's counterfactual use.
H2 · microtasks — chapter two
Coaches don't start with full matches, and piano teachers don't start with the whole piece. They isolate: shoot from close, then dribble, then play. We asked whether the drill structure itself — with no shaping, no hints, no extra reward — is worth anything to a reinforcement learner.
Five agents got identical budgets of 60,000 steps on an 11×7 pitch where only goals are ever rewarded. One played full games from kickoff the whole time. One drilled: 20% of budget spawning near the goal with the ball, 20% spawning in its own half, then full games — the curriculum changes only where episodes start. The controls are where it gets interesting: one agent got uniformly random starts (diversity without structure), and one got optimistic initialization instead of any curriculum at all.
From kickoff, a goal is a long corridor of luck away. The value table waits most of the run for its first win.
Near the goal, wins are cheap — value pools around scoring. The dribble drill then connects midfield to that pool. Full games inherit a table that already knows how stories end.
Time to 90% success on fresh seeds: drilled 15,125 vs whole-game 40,375 (both statistics p≤4e-9; six whole-game seeds never got there). Structure, not luck.
Uniform random starts — diversity without structure — help not at all (43,375; p=0.39 vs whole). And an optimistic-initialization learner needs no drills (16,000). Drills are a remedy for undirected exploration — exactly where theory says instruction pays.
Full-game success from kickoff, 20 greedy evals / 2k steps · 30 seeds · IQM ± 95% CI
Same environment, same reward, same total steps. Drilled curves lift off during the drill phases (shaded). Exploring-starts proves diversity alone is worthless here; whole-optimistic proves a smarter explorer can skip the curriculum. Both boundaries are the ones the guide-policy literature predicts (Xie et al. 2021, JSRL).
EXP6 · no teacher: episodes restart from moments in the agent's own best games · 30 fresh seeds
The founding question of this program: with no teacher, can an agent devise its own microtasks? Yes — restarting from its own best-episode moments beats raw play by 28% (t90 29,000 vs 40,375, Holm p=.02 both statistics). And the teacher still earns their keep: the designed curriculum is another 13,875 steps faster (p=2.9e-7) — a measured price for the generations of compressed experience a teacher carries. The maps show where the self-coach chose to practice: early restarts scatter; late restarts hug the scoring path it discovered.
H3 · variation — chapter three
In 1979, Shea and Morgan split learners into two groups: one drilled each movement pattern in a block, one practiced them shuffled. Blocked practice looked better during training — and lost on retention and transfer. Motor-learning calls it the contextual-interference effect. We reproduced it in a learner small enough to explain.
Three short piano passages share a library of six three-note motifs; each passage has one exception where its fingering breaks the pattern. A linear learner practices either blocked (all of passage A, then B, then C) or interleaved (a random passage each episode). Same total practice. The interference lives exactly where the skill does: in the shared motif weights.
The blocked lane looks immaculate: every rep on the same passage, score climbing. On fresh counterbalanced seeds the practice-room edge is 0.796 vs 0.777 — direction held, no longer significant. The feeling of blocked progress is mostly real recency, and recency fades.
Test everything with learning off. Earlier-drilled passages have decayed under shared-weight drift: retention — interleaved 0.910 vs blocked 0.793 (Welch p=5e-10, order-counterbalanced).
On a never-practiced passage built from the same motifs, interleaved transfers 0.779 vs 0.621 (p≤1.1e-5). Variation forced the weights to encode the motifs, not the last passage rehearsed.
Rebuild the learner without shared motif slots and the entire crossover vanishes (retention gap p=0.92; transfer falls to chance). Interference in shared parameters isn't a story — it's the ablation-verified cause.
Left: score on the currently-practiced passage during training. Right: test with learning off · 40 seeds
With block order counterbalanced across seeds, the v1 "blocked wins the last passage" reversal shrinks to noise — it was an order artifact, exactly as the audit suspected. What survives replication: train-similar, test-different. The Shea–Morgan signature, reproduced and then explained by ablation.
H4 · teachers — chapter four
No pianist learns from scratch. Teachers hand down compressed trajectories — practice this passage, this way — distilled from their teachers, and theirs. The knowledge crosses generations through a narrow channel: not brains, stories. We built the smallest version of that channel we could measure.
A gridworld hides a big reward far to the right, guarded by candy — small terminal rewards near the start that end episodes early and teach agents to stop looking. Each agent lives 15,000 steps with plasticity that decays as it ages: learning rate and exploration halve every 5,000 steps. Old agents remember their three best episodes ever — but can no longer act on them. At death, those episodes become advice: at most 100 state-action pairs, seeded optimistically into a fresh, fully plastic student.
Lives, gorges on candy — and in about one lineage in four, stumbles onto the mountain once, too rigid to consolidate it. Greedy performance at death: candy (0.3), on every seed.
Its advice is not its brain. It's the skeleton of its best days — the one lucky deep run — passed to a student plastic enough to finish the thought.
At 60 fresh seeds the lineage ratchets to IQM 10.0 — 48 of 60 lineages end at the mountain. Weight-copy, both long lives, and no-inheritance stay at candy on every single seed (p≤1.8e-9, both statistics).
Prime a student with 100 random optimistic pairs instead of the teacher's episodes and it does worse than no teaching at all — 0.0, poisoned into loops. The advice works because of what it says, not the enthusiasm it says it with.
One control beats everything: global optimistic initialization solves this small world outright, 60/60. Where blanket optimism is feasible, teachers are redundant — teaching earns its keep where it isn't: big worlds, real fall costs, finite lifetimes.
Greedy return at each generation's end · 60 fresh seeds · IQM ± 95% CI · every baseline comparison Welch Holm p ≤ 1.5e-20
The agents that do find the mountain — 19 of 60 long lives, 16 of 60 first-generation teachers — still do candy at death: they know the way and cannot walk it. The lineage ratchets anyway, to 48 of 60, because each generation's exploration adds new peaks to the next one's inheritance. The dose-matched random-advice control (violet) shows the content matters; the optimistic-init control (red) marks the boundary where no teacher is needed at all. And the decisive identifiability control: fresh students that replayed entire goal-bearing teacher lifetimes consolidated 0 of 35 mountains (dose-matched replay: 0 of 54) versus the distill lineage's 129 of 129 — inheriting raw experience is not inheriting the lesson.
H5 · morphology — chapter five
Toddlers fall about seventeen times an hour and shrug it off — small bodies carry little energy into the ground. By the square-cube law they are also relatively strong: strength scales with s², weight with s³. Falling is how you learn to stand, and evolution priced falling at its cheapest exactly when you need thousands of them.
Our learner balances a torque-limited pendulum whose scale s sets everything honestly: length s, mass 15s³, strength 40s², and fall damage (s)⁴ — impact energy. Seven conditions, 120,000 steps each, all evaluated on the adult body. Some train adult-sized from day one. Some start at half size and grow — on a schedule, or adaptively, growing only when the current body stops falling.
Every early mistake lands at full mass and full height. Damage meter: spinning. It does learn — at a price of 928 damage units to reach competence.
The adaptive grower masters each body before earning the next. Falls happen — at toddler prices. Damage to the same adult competence: 22. Forty-two times cheaper — and a third faster in steps (CI wholly below zero).
The policy transfers across nearby bodies as the dynamics shift smoothly under it — and the ordering survives both physics robustness arms (muscle-torque s³ law, size-scaled damping).
Cumulative fall damage vs training step · 40 fresh seeds · IQM ± 95% CI · damage = (size)⁴ per fall
On the confound-free pairing (growth vs adult, both walking from the start), growing bodies reach adult competence with 9–42× less damage (Welch p ≤ 4.3e-30) and 30–40% fewer steps. In animals, damage is death, injury, and metabolic cost — the currency evolution actually optimizes.
Two of our own sub-claims failed their tests. They stay in the paper.
synthesis
Every mechanism that survived verification worked the same way: it changed which experience the learner got, not how the learner updates. Drills repositioned experience. Variation shuffled it. Teachers curated it across lifetimes. Small bodies discounted its price. And the world model earned its keep not by learning faster but by letting the learner act on experience it wasn't currently having.
That is what "between imitation and evaluation" means operationally. Pure evaluation wastes lifetimes rediscovering what the last lifetime knew; pure imitation caps the student at the teacher. The productive middle is a narrow, curated channel of experience — a drill, an anecdote, a start state — feeding a learner that remains free to disagree. The audit added the boundary: this channel pays where exploration is undirected and mistakes are expensive, which is precisely the condition animals live under. The extension wave closed the loop: an agent can curate its own experience from episodic memory — a self-coach worth 28% — and a teacher's curated experience is worth roughly double that again, which is what instruction is.
Primary comparisons, Holm-corrected within each experiment
methods & colophon
| exp | world | agent | seeds | budget / condition |
|---|---|---|---|---|
| H1 | GridHome 13×11, slip 0.1 | Q, Dyna-Q(20), update-matched ReplayQ; belief dead-reckoning + touch | 30 | 40k steps |
| H2 | SoccerGrid 11×7, p(goal)=1−d/5 | tabular Q; drills, exploring-starts & optimistic controls | 30 | 60k steps |
| H3 | 3 passages · 6 shared motifs · 8 keys | linear Q ± shared features; counterbalanced orders | 40 | 450 episodes |
| H4 | TrapGrid 15×11, candy 0.3, goal 10 | Q, plasticity decay; random-advice & optimism controls | 60 | 5×15k steps |
| H5 | Pendulum, l=s, m=15s³, τ=40s², dmg=s⁴ (+s³ torque & damping robustness arms) | tabular Q, 765 states × 5 torques | 40 | 120k steps |
| E6 | SoccerGrid — self-coach: restarts from own top-10 episodes | tabular Q; whole / teacher / self / self-annealed | 30 | 60k steps |
| E7 | 5 fresh TrapGrids + 5 fresh home pairs (generated, guarded) | headline pairings replicated per layout | 20/15 | as originals |
Statistics: IQM point estimates; 95% percentile bootstrap (10k resamples); two-sided Mann-Whitney AND Welch t, Holm-corrected within each experiment's primary family; censored time-to-threshold assigned budget+1 with censoring fractions reported. Confirmatory runs on fresh seed ranges disjoint from every seed ever used for tuning. Budgets matched to the step; evaluation identical across conditions and excluded from budgets. Design amendments are versioned in DESIGN.md; every v1 claim that failed under audit is reported above, not deleted.
Everything is reproducible from the repository: environments, agents, experiment scripts, 321 tests, this page, and the paper. One command reruns any experiment; the data behind every chart on this page is the committed output of those runs.