A structured prior, the scaled run and the learned belief

Log · 2026-09-19

A structured prior over initial placements

Hypothesis, stated before any code or game (study setups/structured-prior, theory T9 of the program, brief agents/briefs/setup-prior.md): with the same move policy in both seats, a structured setup source, Flag on the back two rows with Bombs likely beside it, Scouts forward, Miners back, Marshal and General apart and the Spy near the General, beats uniform setups on a per-seed paired contrast with the 95% interval above zero for greedy, rollout and expectimax:{"samples":16,"depth":2} under the training rules, and the gain is larger for the players that attack early. The structured source keeps more than 3 bits of entropy on the Flag's square, and a null cell (uniform against uniform on the same seeds) has an interval covering zero. Exploratory motivation, not evidence: on 1,200 unregistered games a front-row Flag lost 61.5% of sides against 32.7% on the back row.

Result: supported on all three policies

The sampler is Setup::structured in the engine (constructive placement, sixteen bounded retries, uniform fallback; 10,000-sample property tests cover every rule, determinism and stream separation), the setup parameter is one WithSetup wrapper applied in armies_search::policy and listed in every catalog schema, and the rules table lives in the server's docs/setups.md. Smoke runs of four games per cell (throwaway seed 900) verified the plumbing before registration: the structured contestant's Flag sat in the back two rows and paired seeds shared its setup across seats. Smokes are not evidence.

Four registrations ran under the training rules, 400 paired games each on seeds 0 to 199, --threads 8, each cell alone on the machine (report):

greedy: structured 71.5% effective (0.669 to 0.757), contrast 0.430 (0.339 to 0.521). Record
rollout: structured 81.5% (0.774 to 0.850), contrast 0.630 (0.551 to 0.709). Record
expectimax sixteen worlds depth 2: structured 83.5% (0.795 to 0.868), contrast 0.670 (0.589 to 0.751). Record
null, uniform against uniform: 48.4% (0.435 to 0.533), contrast minus 0.033 (minus 0.143 to 0.078), covering zero as registered. Record

The decision rule is met for all three policies. Diversity over 10,000 samples per source (setupstats binary, scripts/setup_stats.py): Flag-square entropy 4.14 bits structured against 5.32 uniform (the rule asks for 3); a Flag with no adjacent Bomb in 0.3% of structured setups against 57.1% of uniform ones. Early endings: the null cell ended within 20 plies in 7.2% of games, split evenly; the treatment cells in 4.0%, 7.2% and 6.2%, and all seventy of those games were lost by the uniform side. The brief expected the largest gain for the early attackers; the measured gradient runs the other way (greedy 0.43, rollout 0.63, expectimax 0.67), which the report reads as search converting a setup edge more reliably.

Deviations and failures: none against the brief. The diversity stats use the small arena binary the brief allows, because the Python bindings do not expose setup sources. The cells were re-run twice for provenance: once after the server commit, and again after rebasing over the merged kluss branch (the registry conflict resolved by wrapping its arm with the same setup parameter). Every re-run reproduced the game outcomes exactly; only wall-clock timings moved, and the records say which runs shared the machine. The pre-existing tests/training/test_runlog.py::test_resume_equality_cpu flake (residual CPU nondeterminism documented yesterday by the run-layout study) failed once on this worktree before any change of mine; unrelated to this study. Whether structured becomes the registry default is left to the user, as the brief instructs.

Curated games: seed 0 from both seats of each treatment cell under records/games/structured-v-uniform/ (6 records, indexed and replayed). The compiled armies Python module was not rebuilt; no server/crates/python change.

Attribution: baseten/moonshotai/Kimi-K3 through opencode.

The scaled run is registered

Hypothesis, stated before any measurement: of the three candidates (the small preset, a medium one at embedding 256 and 8 layers over 1,536 environments, and the paper's per-GPU numbers), the largest whose iteration is under three minutes is the one the multi-day run takes, and after 48 hours its EMA move network beats greedy on 200 paired games, after 96 hours expectimax at sixteen worlds and depth two, with the Elo-against-time curve the result either way. This is study ataraxos/scaled-run, theory T5 of the program; the training run itself is launched by the user, never by this agent.

Sizing, measured under the GPU lock on 2026-09-19 with scripts/size_presets.py, three iterations per candidate from scratch, under the day's load (load average 2.4 to 9.9 of 32 threads): small 31.5 s per iteration (25.5, 32.6, 36.4) at 5.96 GiB and 10,350 collection steps per second; medium 76.4 s (63.7, 72.1, 93.4) at 9.51 GiB and 5,970 steps per second; large, the paper's numbers, 135.4 s (95.2, 168.0, 143.1) at 12.14 GiB and 3,900 steps per second. All three sit under three minutes, so the largest qualifies and PRESETS["scaled"] takes the paper's per-GPU numbers with the measurement in its docstring. At that pace the run reaches about 1,280 iterations by hour 48.

The draw rule is confirmed as the paper's, not changed: the training rules preset carries exactly Appendix D Table 17 (a draw after 100 consecutive battleless moves and after 4,000 moves, Rules::TRAINING in crates/engine/src/state.rs), and the advantage filter is on (top quantile 0.75 with a 0.01 magnitude floor). The smoke run's battleless stalling reads as an early-training artefact, so no change to the objective is proposed; the draw share is logged per iteration and is a first-class diagnostic beside entropy and clip rate (a share above half sustained is the registered disproof's second clause).

Both milestones are registered and nothing runs yet: the 48-hour match against greedy and the 96-hour match against expectimax, 200 paired games each (both seats per seed, matched setups, seeds 2000 to 2099, training rules), decided by the per-seed contrast with its 95% interval above zero. scripts/milestone.sh RUN HOUR OPPONENT finds the checkpoint whose run wall clock is nearest the hour, plays the registered games, writes runs/<uuid>/ output and files the record; it reads the registration, so the match cannot drift from it, and its --smoke mode plays throwaway games and files nothing.

The dry run (runs/scaled-dry, kept) proved the schedule end to end: launched through just launch as the systemd unit armies-scaled-dry, four iterations of the scaled preset with --evaluate-every 2 --eval-games 4 wrote checkpoints at every even iteration and evaluations against greedy and expectimax to evaluations.jsonl; stopped by signal mid-run it marked itself stopped; resumed, it finished, and a second resume with the schedule flags repeated evaluated again at iteration 6. Six iterations took 868 s of run wall clock at 133 to 180 s per iteration, draw share 0.61 to 0.67 early in training, and the expectimax evaluations ran through the new Python search bridge. The milestone smokes against both registrations played four throwaway paired games each and filed nothing.

Failures and deviations, all recorded: the run-layout study's CPU resume-equality test flaked before any of my changes (a policy_loss delta of 1.1e-5 against its 1e-5 tolerance, while training/README.md documents up to 5e-4 of per-process CPU noise); its float tolerance is widened to the documented bound in a separate commit. The Python bindings could not play expectimax, which the brief's launch and milestone commands need, so armies.RustPolicy now builds from the full policy registry (armies_search::policy, a two-line server change) and the shared armies module was rebuilt; the learned-belief agent works in the same crate and rebuilt the shared module twice during the session, so my GPU jobs re-verified the bridge before each launch and the merge will carry both changes. A resumed run keeps its hyperparameters but not its schedule flags, so the resume command must repeat --save-every and --evaluate-every; the first dry-run resume omitted them and silently evaluated nothing, which is how it was found. just watch -- -c watch.toml never worked (the recipe is positional, so -c became a recipe name); the recipe now passes any .toml argument as a config, and docs/monitoring.md says so. The dry-run's second resume had to be re-polled after a stale status.json made the first wait exit early; no training time was lost.

The commands the user runs are in the report: launch, watch, both milestones, and the resume.

Attribution: baseten/zai-org/GLM-5.3 through opencode.

The learned belief against the counting prior

Hypothesis, stated before any measurement (study subgame-solving/learned-belief, theory T3 of the program): a belief network trained on the small preset's self-play assigns a higher log-likelihood to the true types of hidden opponent pieces on held-out games between other players (the replay library's greedy, expectimax, rollout and random games) than the counting prior the observations already carry, and expectimax at 16 worlds and depth 2 that samples its worlds from the network beats the same expectimax sampling from the prior on 200 paired games under the training rules. If only the match holds, the belief fits the searcher's own style; if only the likelihood holds, the search does not use what the belief knows. Both outcomes are filed. The disproof is no gain in either measure.

Method, fixed in advance: the move network comes from thirty small preset iterations from seed 0; the belief network trains on its self-play (400 collection steps over 256 environments, 2 epochs). The likelihood measurement replays every game in the replay library through the engine and scores, at every ply of the side to move, the log probability of every hidden opponent piece's true type under the counting prior's belief planes and under the network's marginal estimated from 64 sampled assignments, floored at 1/128 so a zero count is not minus infinity. The match plays 200 paired games, both seats, the same two setups drawn per seed, the candidate sampling 16 assignments from the network per move and the control sampling from the prior, both searching depth 2. The decision rule is the per-seed contrast with a 95% interval above zero.

A sampling bridge is needed first: the Python bindings gain a Searcher wrapping the expectimax search, seeded for determinism, that either samples worlds from the counting prior or searches worlds handed to it as hidden-type assignments. Positions from games the belief policy itself played are not held out anywhere in this measurement; the replay library holds none of its games.

Result

Both measurements are negative (report, record, registration d72d5305). The held-out likelihood, on the library's 64 games, 10,588 positions and 270,223 hidden pieces: the counting prior 2.096 nats per piece and 24.2% top-1, the network 2.293 and 21.4%, worse in every phase and for every acting player. The match on seeds 0 to 99 from both seats: expectimax with belief-sampled worlds 90 wins, 1 draw, 109 losses (45.25%, interval 38.5% to 52.2%) against prior-sampled expectimax, per-seed contrast minus 0.095 (minus 0.217 to 0.027), 196 of 200 games ending by Flag capture. The decision rule is not met; the interval overlaps zero with a negative point estimate, which is inconclusive, not equivalence. The belief side cost 52.9 ms per move against the control's 4.3, both measured while other agents used the machine. On its own self-play the network trained to 1.62 nats per piece, so it fitted a policy thirty iterations old rather than a transferable belief. The study closes at this rung; the retest belongs to the scaled run's checkpoint, and the bridge (the Searcher class, a search-crate test that the sampled and the given-worlds entry points agree bit for bit) is on main for it.

Failures and deviations. The small-preset training crashed at iteration 21 with an illegal action: the setup network sampled a legal setup whose frontier row is Bombs and the Flag with the movables facing lakes, the side to move had no legal move at ply 0, and the engine ends a no-move game only after a move is applied, so the collector stepped an empty legal mask and went uniform over all 10,000 actions. The crash was reproducible from the iteration-20 checkpoint; the fix filters such setups out of the self-play pool (first_mover_can_move in armies_train.selfplay, with tests pinning the engine behaviour), the run resumed from the checkpoint, and the trajectory from iteration 21 differs from an unguarded uninterrupted run. BeliefNet.sample had a second latent bug this study tripped on: decoding in row-major order could spend its movable types before reaching squares that have moved, emptying the whole feasibility mask and degenerating the softmax to uniform, which the engine's assignment validation rejected; the sampler now keeps a lookahead so a completion always exists (armies_train.networks, test tests/training/test_belief_sample.py). The shared venv armies module was overwritten twice by other agents' just build-env runs while this study used it, so every Python here ran against a private install of the module built from this worktree (.armies-module, on PYTHONPATH); the module was also rebuilt once with just build-env as the brief asks. The optional per-game GameRecordDto records were not written. The first launch of the training died to a tool timeout after two minutes; it and all later GPU jobs ran detached with setsid under the lock.

The training log: 30 small-preset iterations at about 10,300 steps per second and 42 seconds each (the throughput agent's attention fix), the belief network 6,394,124 parameters on 102,400 positions in 2 epochs.

Attribution: baseten/zai-org/GLM-5.3 through opencode.