AI & Machine Learning / Case study
Fleet watchdog
A confidence-gated classifier that tells us which AI agents are really working
In short
A hosted decision model classifies each agent pane. Its answer is used only at 0.8 confidence or higher, and the fleet's own status is the fallback.
- State
- Built, in evaluation
- Confidence gate
- 0.8
- Eval size
- 20 cases per run
- Built with
- Node.js / Python
The brief
The problem
We run coding agents in parallel terminal panes, and our board decides what needs attention from the state each pane reports. That state was unreliable. A pane running a long command was tagged idle, the same tag as a pane with nothing to do or one stuck on a rate limit. A board that shows false idles hides stuck work and invites someone to hand a busy agent a second task.
What we did
Our approach
We ran two small labelled evals of 20 cases each. The first compared Jev, a hosted decision model from TypeSafe, with a general chat model. The second gave Jev and Laya, an open reproduction running on a local GPU, the same 20 cases. From those two runs we chose Jev and set the confidence gate. The watchdog now looks only at the end of each pane, asks again only when the pane has changed, and works inside a spending cap. It accepts a classification at 0.8 confidence or higher. Anything else, including an outage or a spent budget, keeps the fleet's own status and marks the row uncertain. The model never types into a pane or retires an agent.
Features
What it does
- 01
Confidence gate at 0.8, with the fleet's own status as the fallback
- 02
Asks again only when a pane has changed
- 03
Shared spending cap across every pane
- 04
Display-only: never sends keystrokes or retires an agent
- 05
Tested against confidence, cache and failure paths
Results
What happened
- Two small 20-case evals chose the model and set the 0.8 confidence gate
- When the model is unsure or unavailable, the board falls back to the fleet's own status instead of guessing
Questions
About Fleet watchdog
Why not use a general chat model?
A decision model returns a label and a confidence in one call, and the confidence is what the gate needs.
Why not run the open model locally for free?
On the same 20 cases its answers did not reach the confidence our gate requires, so every call would have fallen back to the fleet's own status. The community build we used may be missing the model's native decision head, so this is not a verdict on the model. We kept the eval to re-test it.
How big was the eval?
Small: 20 labelled cases per run, 15 pane snapshots across five states plus five triage texts. It was enough to choose a model and set a gate. It does not measure production accuracy.
Services
What this work shows we do
AI automation
Service pageModels wired into the tools your team already uses, with logs and limits.
Fractional CTO
Service pageTechnical decisions, hiring and delivery oversight, part time.
Related work
Built with the same services
- Developer Tools
Agent delivery harness
Every change on this page went through the same pipeline: a board item becomes an agent working in its own isolated copy of the code, then a deterministic gate, then a review by a model from a different vendor, and only then does it land.
Read the case study 
Developer ToolsJarvisMCP
JarvisMCP gives AI agents one governed door into our company systems: CRM, search intelligence, infrastructure, memory and coordination. Agents find the capability they need, act through it, and leave a record. They never hold the keys.
Read the case study
How we work
How a change reaches production
The same five gates for our own products and for yours. Nothing reaches your customers on the strength of a demo.
- 01 · Isolate
Its own branch and workspace
Every change starts in a separate copy of the code, so work in progress never touches what is live.
git worktree add .worktrees/your-change - 02 · Build
Engineers with AI agents
Agents write and test code beside us. They hold no passwords: every call they make goes through our gateway, which checks it and logs it.
agent → gateway → policy · audit · call - 03 · Prove
Evidence, then review
Tests, before-and-after screenshots of the running app, then independent reviewers. It does not move on until no serious finding is left.
review: 0 high · 0 medium → pass - 04 · Ship
Deployed and checked live
The pipeline builds and deploys. Then we check the running service itself, not just a green build.
replicas 1/1 · image sha matches · health 200 - 05 · Watch
Checked every night
Each integration we run is tested against its provider every night, so a changed API is caught before your team notices.
nightly: each integration vs its provider