Records
Throughput of the training loop: one stage dominates, one fix holds
Registration 5dc645d7-0c8a-4491-9de6-f5180a396310 (study pipeline/throughput), run 2026-09-18 on AMD Ryzen AI Max (Strix Halo), 32 threads, 125 GB unified memory, Radeon 8060S via ROCm, Linux 6.18. Server commit 9977d3b (untouched by this study), research commit 9882b10. Every GPU job ran under flock runs/gpu.lock for well under thirty minutes. Other agents' CPU jobs ran during some runs; results are deterministic per seed, so load moves wall time only, and the profile tables below come from single back-to-back sessions.
Commands:
cd research
PYTHONPATH=training .venv/bin/python -m armies_train train --preset small --iterations 3 --out runs/throughput-baseline --seed 0
PYTHONPATH=training .venv/bin/python scripts/profile_collect.py --path reference --out runs/profile-reference.json
PYTHONPATH=training .venv/bin/python scripts/profile_collect.py --path fast --out runs/profile-fast.json
PYTHONPATH=training ARMIES_COLLECT_PATH=reference .venv/bin/python -m armies_train train --preset small --iterations 10 --out runs/throughput-ref-seed0 --seed 0
PYTHONPATH=training ARMIES_COLLECT_PATH=reference .venv/bin/python -m armies_train train --preset small --iterations 10 --out runs/throughput-ref-seed1 --seed 1
PYTHONPATH=training .venv/bin/python -m armies_train train --preset small --iterations 10 --out runs/throughput-fast-seed0 --seed 0Stage table (202 steps of the small preset, synchronize brackets)
| Stage | reference ms/step | share | fast ms/step | share |
|---|---|---|---|---|
| move network forward | 215.27 | 95.7% | 87.60 | 88.8% |
sampling (log_softmax + multinomial) | 2.79 | 1.2% | 2.83 | 2.9% |
| env.observe (Rust) | 1.53 | 0.7% | 1.82 | 1.8% |
| env.observe H2D | 0.92 | 0.4% | 1.05 | 1.1% |
| env.legal_mask (Rust) | 0.78 | 0.3% | 0.99 | 1.0% |
| env.legal_mask H2D | 0.31 | 0.1% | 0.34 | 0.3% |
| obs bf16 store | 0.30 | 0.1% | 0.31 | 0.3% |
| logp/value/acting bookkeeping | 0.27 | 0.1% | 0.32 | 0.3% |
| env.step (Rust) | 0.66 | 0.3% | 0.82 | 0.8% |
| env.step copies | 0.22 | 0.1% | 0.28 | 0.3% |
| finished-game bookkeeping | 0.87 | 0.4% | 1.30 | 1.3% |
| Python remainder | 0.97 | 0.4% | 1.03 | 1.0% |
| total | 224.89 | 98.69 |
Environment share (env.step + env.observe + env.legal_mask including their copies): 4.42 ms of 224.89 = 2.0% before, 5.30 ms of 98.69 = 5.4% after. Engine alone, same sessions: legal_mask + step at 734,623 and 624,192 env steps/s (no Python action picking), observe at 1.05-1.12 M observations/s; the loop's ceiling from the environment side is two orders of magnitude away.
Steps per second (ten iterations, small preset, 206,848 transitions per iteration)
| Run | Path | Mean steps/s | Steps/s, iterations 5-10 | Iteration wall | Total wall |
|---|---|---|---|---|---|
| throughput-ref-seed0 | reference | 4,231 | 4,372 | 73.5 s | 735 s |
| throughput-ref-seed1 | reference | 4,341 | 4,271 | 70.2 s | 702 s |
| throughput-fast-seed0 | fast | 10,150 | 10,129 | 35.0 s | 350 s |
Baseline three-iteration run before any change: 4,133 / 3,601 / 4,291 steps/s, collection 48-57 s of each 57-71 s iteration. The measured factor is 2.4x on collection steps per second (10,150 / 4,231 on run means, 2.28x on the instrumented profiles) and 2.1x on iteration wall time; the update phase speeds up too because the same attention fix applies there (232 s to 140 s of move-update time over ten iterations).
Matched curves (maximum per-iteration absolute difference)
| Metric | fast seed 0 vs reference seed 0 | reference seed 1 vs seed 0 | within the two-seed envelope |
|---|---|---|---|
| policy loss | 0.0088 | 0.0103 | yes |
| entropy | 0.042 | 0.031 | slightly over |
| kept transitions | 13,991 | 10,333 | no (early iterations) |
| games finished per iteration | 72 | 165 | yes |
| mean plies | 18.0 | 87.9 | yes |
Iteration 1 is nearly indistinguishable (entropy 3.030 both, kept 8,699 vs 8,662, mean plies 86 both). The kept difference comes from the advantage filter's 0.01 floor: flash-versus-math rounding (worst logit difference 0.0078 over twenty real batches, one to two bf16 ulps) moves more positions across the floor in iterations 2-6; the runs reconverge by iterations 8-10 (36,980 vs 39,055 vs 36,958). Endings after ten iterations, fast vs reference seed 0: flag captured 1,188 vs 1,227, battleless limit 2,043 vs 1,823, no legal moves 22 vs 29.
Measured and rejected
torch.compileof the move forward: 60.9 ms vs 86.1 ms eager with the flag (a further 1.42x), 36.6 s to compile cold, about 2 s from a warm inductor cache. Rejected: the compiled graph returns NaN rows on real batches at batch 1024 (efficient 44/202 steps, math 50/202, flash 51/202, encoder-only 99/202,split_reductions=False58/202; fp32 or bf16 inputs alike), while the eager path is clean everywhere tested (0/202 steps, zeroed and replicated batches included). A guarded variant caught every NaN but repaid the win in eager recomputes.- Gumbel-max device sampling instead of
multinomial: 2.9 ms/step eager against 2.8 ms; only worth it fused bytorch.compile, which is rejected. Keptmultinomial. - Larger environment counts (compiled forward): 16.8k env forwards/s at 1,024, 15.8k at 2,048, 13.4k at 4,096. The GPU is already saturated at the preset's 1,024; more environments do not help.
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1alone (the shipped fix), same loop otherwise: 97.31 ms/step, 10,523 steps/s against 235.11 ms/step and 4,355 steps/s flag-off in the same session. The flag is what the package now sets by default;ARMIES_COLLECT_PATH=referencereproduces the old loop exactly (verified bit-identical smoke stats, seed 123: policy -0.0056, entropy 2.681, kept 523).
Provenance
Archived raw output (kept out of git): runs/throughput-baseline/history.jsonl (SHA-256 f0d2236cb3f8b50730488c196c236b65a4d36ad9a731790c6e95bcad58205b95), runs/throughput-ref-seed0/history.jsonl (c464f37e8a4d2286c6f2997369d5e53485aa9c9f002c7612b105414268ec1d48), runs/throughput-ref-seed1/history.jsonl (e42563a9dd394b3ed0388ebf6e73b49a17d64d8c5e751ec24e980d7553cded91), runs/throughput-fast-seed0/history.jsonl (7456431e81af5c9c80502af91666faf0ee841e89e7df1a9fc431b04a10da70b3), runs/profile-reference.json (94d2a26a50366ae73db03e84fa2ff53479a0d2a4b83fcefa0723a57443ab42d9), runs/profile-fast.json (8f8f0a2e6b8d299811d6068561dd227c0e3698e93b6aeb55e7550c03ea684a75), runs/profile-reference-aotriton.json (ef3a8f1bfb893480d0724dd416ee5655ee20c0131ed713b3ce8e66b4694b0ce8). Copies of the small files sit next to this record. Runs planned: two profiles, three ten-iteration trainings, one three-iteration baseline; all completed, none interrupted. One earlier fast-path ten-iteration run (runs/throughput-fast-seed0 from the compiled-collect build) trained on zero kept transitions because of the inductor NaN bug and was discarded and rerun; it is described under "Measured and rejected".
Deviations from the registration: the registration predicted 40,000 steps per second from two fixes; the profile showed one stage at 96% and exactly one safe fix exists, giving 2.4x. The study TOML records the measured factor as the outcome, as the approach page's decision rule provides.