Measuring AI agents
We checked our own agents against production records, tested what might replace our grader, and measured what a harness costs. Each note reports what held up and what didn't.
Read them in order. Short on time? Start with note 2, the audit. The later notes lean on it.
- Note 180%of a session on the old agent screen was reading, by our estimate
We Turned AI Agent Management Into a Game Where Levels Control Real Settings
Levels gate real settings, though only tests have proven it so far. Nobody has played an agent up to a level yet.
- Note 276runs our eval passed and an audit failed, of 149 it decided
Our Agent Eval Passed 76 Runs That an Audit Failed
The eval agreed with the audit on 66 of those 149 runs. The judge built to replace it reached 61.5% against an 85% bar.
- Note 364.4%Jev as a judge of agent reports, against the same 85% bar
What Are Decision Models? What We Learned Putting Jev to Work
Jev does fine on narrow labels. As a judge it cost about 111 times less per case and fell short. Version 1.14 scored 56.7%, so we stayed on 1.13.
- Note 41.9xwhat the harness itself added on this job
Agent Harness vs a Single LLM Call: Why Our $0.875 Job Dropped to $0.0016
The 500-times gap compared two different architectures. Most of it probably came from moving the fetching into code.
- Note 546%of reports scoring 85 or higher failed the audit
We Gated Agent Upgrades on a Score. 46% of Top-Scoring Reports Failed Our Audit.
The promotion test caught real bugs. It also optimised towards a number that was a weak guide to whether the agents were right.
See one of these agents
Night Owl. The public page for News Desk, the news agent in note 4. It shows the latest edition, what it cost and past runs.
Open Night Owl