Concepts

Dynamic damping

The paper's central claim is that self-play reinforcement learning fails under imperfect information not because policy gradients are the wrong tool but because their learning dynamics are undamped. Self-play in a game with mixed equilibria tends to cycle: each update over-corrects toward exploiting the previous policy, which the next update punishes. Ataraxos damps this with two coordinated dials, regularisation strength and update size, and moves both along a schedule tied to how strong the policy has become.

The schedules

The reverse KL coefficient toward the magnet policy is for the move network and as the entropy temperature for the setup network, where is the training iteration. The move network's Adam learning rate is , which starts at the upper clip and decays as a power law. The setup network's learning rate stays at .

1e-11e-21e-31e-41e-51e-6110100500training iteration t (log)coefficient (log)setup entropy α = 0.1 / t^0.3move magnet KL = 0.05 / t^0.3move learning rate = clip(0.5 / t^1.1, 5e-6, 1e-4)clipped at 1e-4 until t ≈ 2,300
Both regularisation coefficients decay as the 0.3 power of the iteration while the learning rate decays faster, so late training takes small steps toward a barely regularised target.

Two mechanisms

The first mechanism is that update size should match regularisation strength. A strongly regularised objective has a well-conditioned optimum near the magnet, so large steps are safe and productive; a weakly regularised one has a sharp, moving optimum, so steps must be small to avoid the cycle. Keeping the two commensurate is what turns cyclical or chaotic dynamics into steady progress.

The second is that both should track policy strength. Early, the policy is poor and far from any equilibrium, so aggressive updates under strong regularisation make fast gains. Late, the policy is strong and the remaining improvements are subtle, so the regularisation that would otherwise cap its strength is relaxed while the steps shrink. Annealing too cautiously leaves the model underdeveloped; annealing too aggressively produces fast early gains and then an entropy collapse that leaves the model exploitable and unable to learn further. The authors compare the regularisation to an energy reserve that must be spent at the right rate.

Four brakes on the update

The paper lists four complementary controls of update size, all in use at once: the reverse KL penalty to the data-collection policy (coefficient 0.1), importance-ratio clipping at 0.2, gradient-norm clipping (0.5 for setups, 0.267 for moves), and the learning-rate schedule. Scheduling the learning rate was found to be crucial both for rapid early learning and for avoiding plateaus later.

Entropy as the diagnostic

The training logs in the paper's appendix show the setup and move entropies descending smoothly over the run, never collapsing, while the KL to the data-collection policy and the clip rate fall together. That pair of curves, entropy and clip rate, is the diagnostic the training package prints every iteration here. A run where entropy drops sharply while the clip rate rises has annealed too fast.

What is different from DeepNash

DeepNash damps by a different route: it changes the game rather than the optimiser, adding a regularisation term to the reward and running replicator dynamics to a fixed point before moving the regularisation target. The DeepNash page sets the two side by side, and the DeepNash approach page proposes the comparison under equal compute.