Reference

Monitoring

armies-watch is one terminal for everything long-running: a training run's curves, its latest evaluation, the GPU and CPU behind it, an arena match's progress and the delegated agents working alongside it. It reads files only, so it never touches a run and works on a machine where the run finished hours ago.

just watch ""            # every run updated in the last day, plus gpu, system, agents
just watch runs/latest    # one run: training, evaluations, events, logtail, gpu, system
just watch -- -c watch.toml   # panels from a config file

Install once with the watch extras, which add textual and rich to the shared virtualenv: .venv/bin/pip install -e '.[watch]'.

The keys

q quits, r refreshes every panel, tab and shift+tab cycle focus, 1 to 9 focus a panel by its number, f toggles fullscreen of the focused panel and ? shows this list in the interface. Each panel refreshes on its own interval in its own worker, so a slow file or a stalled rocm-smi never blocks the rest.

The built-in panels

Monitor	Reads	Shows
`training`	a run's `status.json`, `metrics.jsonl`, `run.toml`	the state line, the last metrics row, sparklines over the last 60 iterations for entropy, clip rate, policy loss, steps per second and draw share, each with its latest value and trend
`evaluations`	a run's `evaluations.jsonl`	iteration, opponent, effective score with its interval and milliseconds per move, plus a score sparkline per opponent
`events`	a run's `events.jsonl`	the last events, newest at the bottom, coloured by kind
`logtail`	any text file	the last lines, surviving truncation and replacement
`gpu`	`rocm-smi`	utilisation, memory, temperature and power per card; a machine without the command shows a message instead
`system`	the kernel and `ps`	load averages against the thread count, memory, the top five processes by CPU
`arena`	a `--record-dir` directory of game records	games finished, the score so far, mean milliseconds per move, the newest record
`agents`	`runs/agents/*/meta.json` and `events.jsonl`	every delegated session with brief, model, state, elapsed, cost and last step, coloured by state (green running, gold resuming, red failed, grey finished)

Missing metric keys render as blank and unknown keys are ignored, so the panels keep working while a run's writer changes its fields.

watch.toml

layout = 2        # or layout = { columns = 3 }

[[panel]]
monitor = "training"
run = "runs/watch-demo"

[[panel]]
monitor = "logtail"
path = "runs/watch-demo/train.log"
lines = 10

[[panel]]
monitor = "agents"
path = "runs/agents"

monitor is a built-in name or a plugin module:Class; every other key in the entry becomes a constructor argument. An unknown monitor name is rejected before the interface starts, with the built-ins listed.

Writing a panel

A monitor is one class: name is the panel title, interval the refresh in seconds, update() reads its sources, render() returns a rich renderable, and an optional keys() maps keys to callables that work while the panel is focused. The Monitor base class takes and stores any constructor arguments. Save this as plies.py beside your watch.toml:

import json
from pathlib import Path
from rich.text import Text
from armies_watch.api import Monitor


class Plies(Monitor):
    name = "plies"
    interval = 10.0

    def __init__(self, run="runs/latest", **args):
        self.run = Path(run)
        self.last = None

    def update(self):
        log = self.run / "metrics.jsonl"
        lines = log.read_text().splitlines() if log.exists() else []
        self.last = json.loads(lines[-1]) if lines else None

    def render(self):
        if self.last is None:
            return Text(f"no metrics in {self.run}")
        return Text(f"iteration {self.last['iteration']}: "
                    f"{self.last['mean_plies']:.1f} mean plies")


MONITOR = Plies

Then one panel entry, monitor = "plies.py:Plies" run = "runs/watch-demo", puts it on the grid. A plugin module on the import path is written as module.path:Class. The plugin API, the tailers (FileTail, JsonlTail, read_json, which survive truncation, replacement and half-written files) and the sparkline helpers live in training/armies_watch/api.py and sources.py, and the tests under tests/watch/ exercise every built-in against hand-written fixtures.