Fleet watchdog

The brief

The problem

We run coding agents in parallel terminal panes, and our board decides what needs attention from the state each pane reports. That state was unreliable. A pane running a long command was tagged idle, the same tag as a pane with nothing to do or one stuck on a rate limit. A board that shows false idles hides stuck work and invites someone to hand a busy agent a second task.

Our approach

We ran two small labelled evals of 20 cases each. On the first, Jev, a hosted decision model from TypeSafe, beat a general chat model on accuracy and answered about 40 times faster. On the second, Jev and Laya, an open reproduction running on a local GPU, answered the same 20 cases. Jev got all 20 right, each with high confidence. Laya got 15 right and never reached the confidence we require. The watchdog now looks only at the end of each pane, asks again only when the pane has changed, and works inside a spending cap. It accepts a classification at 0.8 confidence or higher. Anything else, including an outage or a spent budget, keeps the fleet's own status and marks the row uncertain. The model never types into a pane or retires an agent.