100-seed sim policy eval — can any policy family engage the
boat?
Pre-reg posts/2026-08-11-prereg-sim-policy-eval-100seeds.md
· v0 MuJoCo sim, seeds 0–99, 30 s horizon, 30-step
replan · primary metric = progress (initial − final
boat→disk distance, cm), success secondary · phase 2 arms per
owner amendment 22:58Z 08-11 (rung arms killed) · commit 8fc486d
· record-only.
er60k
-0.03 cm
success 0% · best-point +0.02
ftrig4k
+0.08 cm
success 0% · best-point +0.52
snap30k
-0.12 cm
success 0% · best-point +0.33
teacher80k
-0.73 cm
success 0% · best-point +0.58
hold
-0.00 cm
success 0% · best-point +0.00
Gates: reset strikes
0 (gate PASS)
· hold floor -0.00000 cm
(gate PASS)
Per-arm summary (100 paired seeds; "moved" =
|progress| ≥ 0.5 cm)
| arm | policy | mean progress |
median | mean best-point | moved | success |
med. success tick | med. latency ms | panel MAE |
|---|
| er60k | er_60k trunk (euler-1 expert, heun-10 flow) | -0.0288 | -0.0000 | +0.0183 | 4/100 | 0% | — | 423 | 5.7782 |
| ftrig4k | rig-ft snapflow student (euler-1) | +0.0819 | +0.0076 | +0.5153 | 47/100 | 0% | — | 63 | — |
| snap30k | snapflow student (euler-1) | -0.1168 | +0.0000 | +0.3277 | 38/100 | 0% | — | 65 | — |
| teacher80k | artrunk 80k teacher (heun-30) | -0.7290 | -0.0411 | +0.5809 | 56/100 | 0% | — | 1183 | — |
| hold | hold control (zero action) | -0.0000 | -0.0000 | +0.0001 | 0/100 | 0% | — | — | — |
Paired per-seed reads (bootstrap CI95, 10k
resamples, seed 0)
| pair | Δ mean (cm) | CI95 |
CI excl. 0 | win rate |
|---|
| er60k − hold | -0.0288 | [-0.0882, +0.0140] | no | 7% |
| ftrig4k − hold | +0.0819 | [-0.1674, +0.3195] | no | 52% |
| snap30k − hold | -0.1167 | [-0.3668, +0.0985] | no | 46% |
| teacher80k − hold | -0.7290 | [-1.1787, -0.3373] | yes | 31% |
| ftrig4k − er60k | +0.1107 | [-0.1457, +0.3581] | no | 53% |
| snap30k − er60k | -0.0880 | [-0.3420, +0.1351] | no | 45% |
| teacher80k − er60k | -0.7002 | [-1.1600, -0.2922] | yes | 30% |
Ordering read: only ['er60k'] present — no pairs to rank (the phase-2 owner amendment killed the rung arms).
Charts
Video gallery (deterministic picks: best / median
/ worst seed by progress)
er60k er_60k trunk (euler-1 expert, heun-10 flow) — the money shot: confident reaching over the table, never at the boat (the visual-gap fingerprint)
best seed 36progress +0.55 cm · no successinitial 9.8 → final 9.2 cm (min 9.2)
median seed 67progress -0.00 cm · no successinitial 10.9 → final 10.9 cm (min 10.9)
worst seed 22progress -2.01 cm · no successinitial 10.6 → final 12.6 cm (min 10.2)
reach-but-miss seed 0progress +0.00 cm · no successinitial 10.5 → final 10.5 cm (min 10.5)
reach-but-miss seed 1progress -0.00 cm · no successinitial 7.3 → final 7.3 cm (min 7.3)
ftrig4k rig-ft snapflow student (euler-1)
best seed 20progress +3.64 cm · no successinitial 9.4 → final 5.8 cm (min 5.8)
median seed 6progress +0.01 cm · no successinitial 10.1 → final 10.1 cm (min 10.1)
worst seed 16progress -6.68 cm · no successinitial 9.7 → final 16.4 cm (min 9.7)
snap30k snapflow student (euler-1)
best seed 83progress +2.78 cm · no successinitial 11.3 → final 8.5 cm (min 8.5)
median seed 10progress +0.00 cm · no successinitial 11.6 → final 11.6 cm (min 11.6)
worst seed 56progress -7.31 cm · no successinitial 11.8 → final 19.1 cm (min 11.8)
teacher80k artrunk 80k teacher (heun-30)
best seed 93progress +3.23 cm · no successinitial 10.3 → final 7.1 cm (min 7.1)
median seed 38progress -0.03 cm · no successinitial 10.4 → final 10.5 cm (min 10.4)
worst seed 27progress -12.42 cm · no successinitial 10.5 → final 22.9 cm (min 6.2)
Generated by fontaine/scripts/sim100_report.py from the
banked analysis JSON + per-arm episode JSONs; every artifact
regenerable offline.