Concepts

Self-play reinforcement learning

Ataraxos is tabula rasa: two transformers, one for setups and one for moves, start random and play against themselves. There is no search in the data-generating loop. The authors tried search-based self-play in the style of AlphaZero and found the slowdown outweighed the benefit, likely because the number of games matters more than their quality when the things to learn are a broad distribution over setups and a policy for each pair of setups.

The loop

Each of 16 GPUs simulated 1,536 games at once and played 202 moves in every one of them between training iterations, about five million transitions per iteration. Setups came from a pool of 1,000 per player per GPU sampled from the setup network and refreshed every iteration. Finished games restart independently, so the boards drift out of phase and the training data covers every stage of the game. Training used bfloat16, which sped iterations up about threefold, and an exponential moving average of the parameters (0.999) for evaluation.

Targets

For a move made at position with subsequent positions and outcome (win 1, draw 0, loss ), the advantage is a -return with over the value predictions , minus the baseline :

where bootstraps from the value own-moves later, or uses when the game has ended. The outcome target is a separate -return with over the three-way outcome probabilities, and the value head is trained by cross-entropy toward it.

Advantage filtering

Only moves whose advantage magnitude is in the top quarter and at least 0.01 enter the training batch. This cut the wall-clock time per iteration by about 2.5 times and, unexpectedly, improved both sample efficiency and final strength. Without it, move entropy rose and learning slowed.

The policy loss

With the importance ratio to the policy that generated the data, the move loss is

The first term is the clipped surrogate of PPO. The second is a reverse KL to the data-collection policy, a second brake on the step size. The third is a reverse KL to the magnet policy , which picks a movable piece uniformly and then one of its legal moves uniformly; its coefficient anneals over training. The value loss is added with weight one, gradients are clipped at norm 0.267, Adam runs one epoch over the 202 batches, and the learning rate follows its own schedule. The schedules and the reasoning behind them are the subject of dynamic damping.

Setups

The setup network is trained on whole setups from finished games with Monte Carlo returns, no bootstrapping, no filtering. Its advantage adds an entropy term: the difference between the realised conditional entropy of the setup given a prefix and the network's own prediction of it, weighted by the same annealed . The authors remark that Monte Carlo returns beating bootstrapped ones is unusual in RL and has also been seen in language-model reasoning, perhaps because both are bandit-like: one decision, then a long noisy rollout. The setup network page covers the architecture.

What this program does with it

The training package implements this loop against the Rust environment bindings with the paper's hyperparameters as defaults and a small preset for smoke runs. The first registered run is twenty iterations of the small preset, evaluated against random and greedy; that is a pipeline test, not a strength claim. The scale question, how far one GPU and a week can get, is the subject of the Ataraxos approach page.