Approaches

The pipeline on one machine

How much of this workstation can the training loop and the search use, and can a multi-day run be watched and resumed without an agent in the loop?

The mechanism

Three pieces. A run layout: every training run writes metrics.jsonl, status.json and periodic checkpoints under runs/<id>/, resumes from the last checkpoint, and is launched by a command that survives the terminal. A monitor: a terminal interface that tails those files and the machine's counters, with monitors defined as small plugins so that a new view of a long run is one file. A throughput study: a profile of the collection loop and the fixes it justifies, measured as environment steps per second with unchanged learning curves.

The plan

  1. run-layout: the file contracts in ../../docs/plan.md, resume, the launcher recipes, and a smoke run that stops and resumes.
  2. monitor: the armies-watch package, its plugin API, built-in monitors for training curves, evaluation checkpoints, GPU and CPU load, arena progress and delegated agents, with tests that feed recorded files.
  3. throughput: the profile table before and after, with the learning curves of ten seeded iterations matched.

Decision rule

The run layout is done when a run killed mid-iteration resumes to the same metrics as an uninterrupted run from the same seed. The monitor is done when each built-in monitor renders from a recorded fixture in a test and the interface stays responsive on a live run. The throughput study succeeds if collection reaches 40,000 steps per second on the small preset with matched curves; a smaller gain is reported as the measured factor.

What could disprove it

The loop may already be within a factor of two of the network's forward cost, in which case the throughput study reports that and the scaled run is planned at the measured speed.

Evidence

Measured on September 18, 2026 before any change: the small preset collects 4,100 steps per second on 1,024 environments, 60 seconds per iteration, of which about 50 seconds is collection.

The monitoring interface is done. armies-watch follows a training run's curves and state line, its evaluation table, its events and log, the GPU, the machine, an arena record directory and the delegated agents, from files alone; every built-in renders a hand-written fixture in a test (34 tests under tests/watch/), and a live session against a three-iteration smoke run kept refreshing the machine panels while the run trained on the GPU. A new view is one plugin file against the Monitor base class, per the decision rule in this page.

The run layout is done (September 18, 2026, run-layout study): on the GPU, the production path, a run resumed after a checkpoint is bit-identical to an uninterrupted one (full smoke preset, 22 games finished in the resumed iteration, every metric delta zero); on the CPU the trajectories, game counts and endings match exactly and the float metrics within 5e-4, because a rare per-process noise in this PyTorch build's CPU backend survives the pins (oneDNN off, one OpenMP thread) that remove the two larger variance sources. The checkpoint carries the networks, optimisers, setup pool, torch and numpy generators, every environment slot's live game, and a replay of the environment's hidden per-slot generator draws. just launch kept a run alive through the closing of the shell that launched it, and just stop plus --resume finished a run stopped mid-flight. Only random and greedy can play as evaluation opponents through the current Python bindings.

The throughput study filed the same day: the move network's forward pass is 96% of a collection step, the ROCm math-attention fallback is 60% of it, and enabling the build's experimental attention kernels lifts collection to 10,100 steps per second (2.4x) with matched ten-iteration curves; the environment's share of collection wall time is 2.0% before the fix and 5.4% after. Throughput report.