Our Agent Eval Passed 76 Runs That an Audit Failed

Our agents write short reports every day: what they found, what it means and what to do next. Each report went through an automated eval, and most came back green.

One week in October, an agent called Hype spent $24.18 of the $24.45 recorded that week. On October 23 our weekly cap stopped the org's agents. Of our 13 agents, only CFO saw it coming, and even CFO never named Hype. Tinker, the agent whose job is optimisation, said "No change needed".

The agents watching the org sat under the same cap, so they went down with everything else and couldn't report the outage. And their earlier reports had looked fine.

So we stopped asking whether the reports passed and started asking whether they were true.

How we audited them

This work was agent-led. Claude did the grading, and no human graders were involved. For each run, it pulled every claim in the headline and the top three points, then checked each one against records the report's author didn't produce: spend ledgers, run records, gateway traces, GitHub and live URLs.

Each run then got a score out of 8:

A run passes only if Correct is 2 and the total is at least 5. If Correct is 0, the whole run scores 0. The rule is meant to stop a beautifully written report with a wrong number from passing.

We have now done this three times, as we fixed things between rounds.

Add those up and the old eval agreed with the audit on 66 of 149 decided runs. It passed 76 runs the audit failed. These rounds used different runs and different fixes, so don't read them as a trend line. We read them as the same finding three times.

What the claims looked like

Across all three rounds, the audit checked 392 claims:

Flat-out false claims were the smallest group. Misleading ones were more than twice as common, and they do more damage because they're harder to spot. Here's how one false claim went through both graders:

Two more examples from the latest round:

• A number that can't move. Release Keeper reports "errors steady at 0" every day. Its query filters on two fields that none of the 61 stored error documents have. It would say 0 during an outage.
• Weaker sourcing than claimed. News Desk cited "two independent sources". One was a link-blog post about the other.

Results varied a lot by agent. Some scored badly with accurate numbers: Org Brief's figures were exact, but it gave no next step, so it never reached the bar.

Why the eval was wrong

Nothing about this was mysterious once we looked:

1. Judge votes never decided the result. Every judge rubric was marked optional, and the runner ignored optional judges. The LLM judge could say no and the run would still pass.
2. The judge only saw the note. It never saw the facts or yesterday's run, so it couldn't catch a wrong number or a repeat.
3. The fact checks marked their own homework. They compared the note to facts computed by the same run. A wrong cause label passed.
4. The usefulness score was partly a keyword regex. Words like "pause it?" and "You can" scored points, even when the advice was wrong.
5. Format checks failed accurate notes. Some rules cared about wording, so a correct report could fail and a wrong one could pass.
6. Agents couldn't see setup failures. Those stops were dropped from every run count, so agents that had never run disappeared from the reports.

None of these would show up in a dashboard of pass rates. A pass rate tells you how often your checks said yes, and we had built checks that found it easy to say yes.

Then we overcorrected

We built a new judge. It takes three votes per run and gets independent recounts of the records a report quotes, as of the time of the run. We measured it against the audit before letting it decide anything, and we set the bar at 85%.

It reached 61.5% (75 of 122), or 64.1% of the 117 cases it could evaluate. The errors had also flipped direction:

The old eval wrongly passed things. The new judge wrongly fails things, and that happens to other teams too. A recent paper on computer-use agents reported the same pattern: on one of its two datasets, upgrading a judge's model cut false passes from 45% to 10% while false failures rose from 24% to 44%. Improving one error rate doesn't mean you've solved the problem.

So the judge stays in shadow mode. It returned a grade on 26 of the latest 46 runs and was unavailable on the other 20. It decides nothing.

Who audits the auditor?

Our reference grades came from an agent, so an obvious question is whether the reference is any good. So on the 14 cases where we suspected the reference, a fresh agent re-read each run from scratch.

Applying those rulings lifts the judge from 64.1% to 66.7%. Giving it every ambiguous case too reaches 70.1%. Neither gets close to 85%, so the errors we checked don't explain the gap. We re-read only 14 of the 42 disagreements, so others could move the number either way.

The limits still matter. The grader, the judge and the adjudicator all come from the same model family, so they may share blind spots. The adjudicator wasn't fully blind, because it knew which side of a disagreement each case came from. The grader could see the eval's verdict while grading. In one case, the first audit applied a single judgement to three similar News Desk runs. In 22 of 122 reference grades, the audit awarded Correct 2 while recording a claim it didn't mark true. All this tells you the audit is much stricter and better informed than the eval. It doesn't prove the audit is right.

Grading now costs more than the work

Agents got cheap once we cut our most expensive one and moved others to lighter harnesses. Grading didn't.

We think that's the right trade for now. But if you budget for agents, budget for grading them too, and make sure your agents' own cost reports include it. Ours left it out, and we only noticed because the audit compared them.

Where we pick this up

Handover sheet · Nov 1

Our notes for the next shift. If you run agents too, this list may save you a lap.

• Open
  Fix Release Keeper's error countIts query filters on two fields that none of the 61 stored error documents have, so it can only read 0.
  release.py
• Open
  Stop Sentinel calling plan refusals rate limitsIt reported the wrong cause and the wrong fix on three days running.
  gateway_traces
• Open
  Bring down the judge's false failuresIt wrongly failed 39 of 117 cases. Unchecked claims account for 14 of its disagreements with the audit, so start there.
  Office grader
• Open
  Re-read the other 28 disagreementsWe only checked 14 of the 42, and the rest could move the number either way.
  calibration set
• Open
  Put grading in the agents' cost reportsReview and judge calls cost about six times the runs themselves, and CFO left them out.
  cfo_teller
• Waiting
  Give the judge a real voteIt stays in shadow until it reaches 85%.
  FF_OFFICE_GRADE_ENFORCED
• Shipped
  Reserve budget for the watchersWatchers now get reserved headroom inside the cap, and Hype is gone. A real cap hit hasn't tested it yet.
  #3977

Three ideas from this are worth borrowing even if you never touch our code. Check claims against records the author didn't produce. Search your config for judges nothing reads. Measure both error directions, since a single agreement number hid that our two graders fail in opposite ways.

We run this on our own organisations. If you're building agents that act on your data and want to compare notes or try it as a design partner, write to boris@curate-me.ai.