Vol. III · Issue 14 · NOV 2026 · Established MMXXIV
Architecture Deep Dive·November 2, 2026·6 min read
Measuring AI agentsNote 5 of 5

We Gated Agent Upgrades on a Score. 46% of Top-Scoring Reports Failed Our Audit.

A new agent version that scored worse than the best one was thrown away. Then audits of 165 runs found that 46% of reports scoring 85 or higher failed.

BB
Boris BarashBuilder of things with AI. Creator of curate-me.ai.
#llm-evals#ai-agents#prompt-optimization
A small challenger robot and a bigger champion robot stand on a fairground scale while a judge robot reads the result from a floppy bent measuring stick, with a row of podiums climbing the hill behind them
Agent-led
No human graders
AI-generated illustration
Prompt · Codex image generation, Nov 2026

Landscape 3:2 editorial illustration in a risograph print style, two inks (deep navy and warm orange) on warm cream paper, visible grain, generous empty space, clean shapes, calm witty New Yorker-like humour. Absolutely no text, no letters, no numbers, no logos, no brand marks, no real people. Scene: A small challenger robot and a slightly bigger champion robot stand side by side on a weighing scale at a village fair, while a judge robot reads the result from a long measuring stick that is visibly bent and wobbly like a noodle, nodding confidently. A small flag on the champion's platform. Behind them a tall ladder of podiums rises up a hill. No markings on the stick.

AI Collaboration
§ Colophon

Agent-led. Agents ran the research, tests, grading and drafting. No human graders were involved.

Claude (Opus 5.5) via Claude Code, Run matching, recounts, charts and drafting

Codex image generation, Cover, concept illustrations and flowchart

Total AI cost: $14.33

Governed by curate-me.ai

On this page

In October we set a rule for our AI agents: a new version that does worse than the best one we have, on a score out of 100, gets thrown away. We call it the promotion test. Quality was only supposed to move one way.

It worked as designed. It caught dips, refused challengers that newly failed a required check, and served the winning version in production. It also optimised towards a number that, as we later found, was a weak guide to whether the agents were right.

165Runs with both numbersPromotion score and audit grade
46%Top scores that failed37 of 81 scoring 85 or higher
0.42Correlation, one seat aside0.044 with News Desk included

How the promotion test works

Flowchart. Frozen facts, the same inputs for both, go to the champion, the best version so far, and the new version, the challenger. Both get a score out of 100. If the new version is worse, or has a new required-check failure, it's discarded and the champion serves production. If it's ahead by 3 or more, it's promoted and becomes the champion. Otherwise both are kept: the new code stays in production and the champion record stays. Later, the audit found that 37 of 81 runs scoring 85 or higher failed.
The promotion test as it runs: same facts, two versions, one score out of 100. The audit is separate and came later, and it isn't part of the decision.
Prompt · Codex image generation, Nov 2026
Desktop version

A detailed production-quality flowchart, landscape 3:2, in a risograph print style: two inks, deep navy and warm orange, on warm cream paper, light grain. Clean geometric shapes, straight orthogonal thin navy arrows with small heads, consistent line weight, generous spacing, one clean bold sans-serif typeface, large text that stays readable when shrunk. Orange is used on one element only, as named below; any text on orange is navy. No title. Spell every label exactly as written, nothing added, no extra words. The straight double quotes in this prompt only mark where a label starts and ends: never draw them. Two rows, with large text. Top row, left to right: rectangle "Frozen facts" with smaller line "The same inputs for both". An arrow splits into two stacked rectangles: "Champion" with smaller line "the best version so far", and "New version" with smaller line "the challenger". Both arrows join into a rectangle: "Score out of 100" with smaller line "Checks 20, Required 15, Judge 25, Headline 15, Usefulness 25". An arrow runs from the score rectangle down and back to the start of the second row. Second row, left to right: a wide hexagon "Worse, or a new required-check failure?". Its arrow labelled "No" goes right to a second wide hexagon "Ahead by 3 or more?". That hexagon's arrow labelled "Yes" goes right to a rectangle "Promote" with smaller line "It becomes the champion". Beneath the first hexagon, reached by an arrow labelled "Yes": a rectangle "Discard it" with smaller line "The champion serves production". Beneath the second hexagon, reached by an arrow labelled "No": a rectangle "Keep both" with smaller line "New code stays in production, champion record stays". At the bottom, a separate wide banner filled orange, not connected by any arrow: "Later, the audit: 37 of 81 runs scoring 85 or higher failed".

Phone version

A detailed production-quality flowchart for a phone screen, portrait 9:16, in a risograph print style: two inks, deep navy and warm orange, on warm cream paper, light grain. Clean geometric shapes, straight orthogonal thin navy arrows with small heads, consistent line weight, generous spacing, one clean bold sans-serif typeface, large text that stays readable when shrunk. Orange is used on one element only, as named below; any text on orange is navy. No title. Spell every label exactly as written, including punctuation, nothing added, no extra words. Very large text; even the smaller lines must be comfortably large. Top to bottom: 1. Rectangle: "Frozen facts" with smaller line "The same inputs for both". 2. Arrow splits into two side-by-side rectangles: "Champion" with smaller line "the best so far"; "New version" with smaller line "the challenger". 3. Both join into a rectangle: "Score out of 100" with smaller line "Checks 20, Required 15, Judge 25, Headline 15, Usefulness 25". 4. A wide hexagon: "Worse, or a new required-check failure?". An arrow labelled "Yes" goes down to a rectangle "Discard it" with smaller line "The champion serves production". An arrow labelled "No" runs down the right edge to a second wide hexagon: "Ahead by 3 or more?". From it, an arrow labelled "Yes" leads to a rectangle "Promote" with smaller line "It becomes the champion", and an arrow labelled "No" leads to a rectangle "Keep both" with smaller line "New code stays in production, champion record stays". These two rectangles sit side by side. 5. At the bottom, a separate banner filled orange, not connected by any arrow: "Later, the audit: 37 of 81 runs scoring 85 or higher failed".

A round compares versions on the same frozen facts. Each one gets a score from 0 to 100, built from five parts, minus up to 20 points for filler phrases:

What made up the promotion score

Judge votes25%
Usefulness signals25%
Check pass rate20%
Required checks15%
Headline quality15%

A version that scores below the champion, or newly fails a required check, is discarded. One that beats the champion by 3 points or more takes over. In between, the new code stays in production and the champion record doesn't change. We later added model bake-offs, so a seat could try four models on the same facts. Any model within 3 points of the top competed on price, so a winner didn't have to beat anything.

A boxing ring where a referee robot raises a champion's arm while holding up a floppy measuring stick
Prompt · Codex image generation, Nov 2026

Landscape 3:2 editorial illustration in a risograph print style, two inks (deep navy and warm orange) on warm cream paper, visible grain, generous empty space, clean shapes, calm witty New Yorker-like humour. Absolutely no text, no letters, no numbers, no logos, no brand marks, no real people. Scene: A boxing ring seen from the side, with a champion robot wearing a belt shape and a smaller challenger robot in opposite corners, both looking fit. Between them a referee robot raises one arm to declare a winner while holding up a bendy noodle-like measuring stick in the other hand. Small crowd of robots in simple shapes. No letters or numbers.

The rounds found real bugs along the way

The evaluation rounds were useful, and most of what they caught had nothing to do with scores. Two of these fixes landed just before the test itself did:

  • Thinking ate the judge's budget. The judge kept returning invalid responses. Sonnet 5 was spending the entire token allowance on thinking and returning no text, so we turned thinking off for the judge.
  • Tests starved production. About 150 evaluation runs pushed the weekly counter to $24.91 of $25, and real scheduled agents were refused. Evaluation now has its own budget.
  • A stale connection looked like missing setup. An old database handle caused 11 of 15 failed runs on one day. Those seats appeared to need a gateway key when they didn't.
  • An undecided round wrote a winner. Three candidates ran out of budget, and the survivor was briefly set as Tinker's model before we reverted it. Now a bake-off with no comparable result writes nothing.

If we'd stopped here, we'd have called it a success.

Then we compared the score with the audit

Our agent-led audit checks each report's claims against production records and scores it out of 8. To pass, every checked claim has to hold and the total has to reach 5. By November 1 we had 165 runs with both numbers. We plotted one against the other.

Audit failAudit passSquare: News Desk02468020406080100News Desk: low score, high audit85+: 37 of 81 failedScore 88, audit 6 of 8, passedScore 88, audit 5 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 4 of 8, failedScore 75, audit 5 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 8 of 8, passedScore 58, audit 3 of 8, failedScore 60, audit 2 of 8, failedScore 56, audit 2 of 8, failedScore 58, audit 2 of 8, failedScore 58, audit 2 of 8, failedScore 100, audit 5 of 8, passedScore 81, audit 5 of 8, failedScore 81, audit 5 of 8, passedScore 83, audit 6 of 8, passedScore 100, audit 5 of 8, passedScore 75, audit 5 of 8, passedScore 92, audit 4 of 8, failedScore 92, audit 4 of 8, failedScore 92, audit 4 of 8, failedScore 92, audit 6 of 8, failedScore 62, audit 2 of 8, failedScore 62, audit 2 of 8, failedScore 62, audit 3 of 8, failedScore 75, audit 2 of 8, failedScore 62, audit 3 of 8, failedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 100, audit 4 of 8, failedScore 100, audit 3 of 8, failedScore 100, audit 4 of 8, failedScore 33, audit 7 of 8, passed, News DeskScore 21, audit 6 of 8, passed, News DeskScore 45, audit 6 of 8, passed, News DeskScore 41, audit 7 of 8, passed, News DeskScore 46, audit 7 of 8, passed, News DeskScore 17, audit 3 of 8, failedScore 41, audit 3 of 8, failedScore 41, audit 5 of 8, passedScore 12, audit 4 of 8, failedScore 25, audit 4 of 8, failedScore 71, audit 4 of 8, failedScore 20, audit 5 of 8, failedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 7 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 7 of 8, passedScore 100, audit 4 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 70, audit 8 of 8, passedScore 88, audit 7 of 8, passedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 83, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 50, audit 5 of 8, failedScore 34, audit 6 of 8, failed, News DeskScore 34, audit 7 of 8, passed, News DeskScore 34, audit 7 of 8, passed, News DeskScore 58, audit 5 of 8, passedScore 58, audit 6 of 8, passedScore 62, audit 2 of 8, failedScore 62, audit 2 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 70, audit 6 of 8, passedScore 75, audit 3 of 8, failedScore 93, audit 4 of 8, failedScore 80, audit 4 of 8, failedScore 65, audit 0 of 8, failedScore 55, audit 3 of 8, failedScore 55, audit 3 of 8, failedScore 55, audit 3 of 8, failedScore 21, audit 7 of 8, passed, News DeskScore 21, audit 7 of 8, passed, News DeskScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 62, audit 0 of 8, failedScore 75, audit 0 of 8, failedScore 62, audit 0 of 8, failedScore 100, audit 6 of 8, failedScore 75, audit 3 of 8, failedScore 88, audit 6 of 8, failedScore 62, audit 0 of 8, failedScore 70, audit 3 of 8, failedScore 88, audit 7 of 8, passedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 68, audit 0 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 34, audit 6 of 8, passed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 62, audit 0 of 8, failedScore 68, audit 2 of 8, failedScore 88, audit 5 of 8, passedScore 70, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 68, audit 6 of 8, passedScore 70, audit 4 of 8, failedScore 75, audit 5 of 8, passedScore 88, audit 6 of 8, passedScore 28, audit 7 of 8, passed, News DeskScore 33, audit 6 of 8, failed, News DeskScore 21, audit 5 of 8, failed, News DeskScore 85, audit 4 of 8, failedScore 85, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 75, audit 5 of 8, failedScore 75, audit 3 of 8, failedScore 75, audit 3 of 8, failedScore 68, audit 3 of 8, failedScore 80, audit 3 of 8, failedScore 63, audit 3 of 8, failedScore 75, audit 4 of 8, failedScore 58, audit 3 of 8, failedScore 62, audit 3 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 75, audit 6 of 8, passedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 34, audit 6 of 8, passed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 88, audit 2 of 8, failedScore 88, audit 2 of 8, failedScore 75, audit 2 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedPromotion score (out of 100)Audit total (out of 8)Audit failAudit passSquare: News Desk02468020406080100News Desk: low score, high audit85+: 37 of 81 failedScore 88, audit 6 of 8, passedScore 88, audit 5 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 4 of 8, failedScore 75, audit 5 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 8 of 8, passedScore 58, audit 3 of 8, failedScore 60, audit 2 of 8, failedScore 56, audit 2 of 8, failedScore 58, audit 2 of 8, failedScore 58, audit 2 of 8, failedScore 100, audit 5 of 8, passedScore 81, audit 5 of 8, failedScore 81, audit 5 of 8, passedScore 83, audit 6 of 8, passedScore 100, audit 5 of 8, passedScore 75, audit 5 of 8, passedScore 92, audit 4 of 8, failedScore 92, audit 4 of 8, failedScore 92, audit 4 of 8, failedScore 92, audit 6 of 8, failedScore 62, audit 2 of 8, failedScore 62, audit 2 of 8, failedScore 62, audit 3 of 8, failedScore 75, audit 2 of 8, failedScore 62, audit 3 of 8, failedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 100, audit 4 of 8, failedScore 100, audit 3 of 8, failedScore 100, audit 4 of 8, failedScore 33, audit 7 of 8, passed, News DeskScore 21, audit 6 of 8, passed, News DeskScore 45, audit 6 of 8, passed, News DeskScore 41, audit 7 of 8, passed, News DeskScore 46, audit 7 of 8, passed, News DeskScore 17, audit 3 of 8, failedScore 41, audit 3 of 8, failedScore 41, audit 5 of 8, passedScore 12, audit 4 of 8, failedScore 25, audit 4 of 8, failedScore 71, audit 4 of 8, failedScore 20, audit 5 of 8, failedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 7 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 7 of 8, passedScore 100, audit 4 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 88, audit 5 of 8, passedScore 70, audit 8 of 8, passedScore 88, audit 7 of 8, passedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 83, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 50, audit 5 of 8, failedScore 34, audit 6 of 8, failed, News DeskScore 34, audit 7 of 8, passed, News DeskScore 34, audit 7 of 8, passed, News DeskScore 58, audit 5 of 8, passedScore 58, audit 6 of 8, passedScore 62, audit 2 of 8, failedScore 62, audit 2 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 70, audit 6 of 8, passedScore 75, audit 3 of 8, failedScore 93, audit 4 of 8, failedScore 80, audit 4 of 8, failedScore 65, audit 0 of 8, failedScore 55, audit 3 of 8, failedScore 55, audit 3 of 8, failedScore 55, audit 3 of 8, failedScore 21, audit 7 of 8, passed, News DeskScore 21, audit 7 of 8, passed, News DeskScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 62, audit 0 of 8, failedScore 75, audit 0 of 8, failedScore 62, audit 0 of 8, failedScore 100, audit 6 of 8, failedScore 75, audit 3 of 8, failedScore 88, audit 6 of 8, failedScore 62, audit 0 of 8, failedScore 70, audit 3 of 8, failedScore 88, audit 7 of 8, passedScore 88, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 68, audit 0 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 34, audit 6 of 8, passed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 62, audit 0 of 8, failedScore 68, audit 2 of 8, failedScore 88, audit 5 of 8, passedScore 70, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 68, audit 6 of 8, passedScore 70, audit 4 of 8, failedScore 75, audit 5 of 8, passedScore 88, audit 6 of 8, passedScore 28, audit 7 of 8, passed, News DeskScore 33, audit 6 of 8, failed, News DeskScore 21, audit 5 of 8, failed, News DeskScore 85, audit 4 of 8, failedScore 85, audit 4 of 8, failedScore 88, audit 4 of 8, failedScore 75, audit 5 of 8, failedScore 75, audit 3 of 8, failedScore 75, audit 3 of 8, failedScore 68, audit 3 of 8, failedScore 80, audit 3 of 8, failedScore 63, audit 3 of 8, failedScore 75, audit 4 of 8, failedScore 58, audit 3 of 8, failedScore 62, audit 3 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 75, audit 6 of 8, passedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 75, audit 4 of 8, failedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 100, audit 6 of 8, passedScore 34, audit 6 of 8, passed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 34, audit 5 of 8, failed, News DeskScore 88, audit 2 of 8, failedScore 88, audit 2 of 8, failedScore 75, audit 2 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 3 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 4 of 8, failedScore 88, audit 2 of 8, failedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedScore 88, audit 6 of 8, passedPromotion score (out of 100)Audit total (out of 8)
One seat hides the signal in the score: the correlation is 0.044 with News Desk (squares) and 0.42 without it. Shaded on the right: scores of 85 or higher. 165 runs, nudged slightly so overlapping dots show.

Pooled across every seat, the correlation is 0.044. Our first draft of this post stopped there and said the score predicted nothing. A fact-checking agent pointed out that this was wrong, and we confirmed it from the raw files.

One seat was flattening most of it. News Desk scored low on this score and high in the audit, and it wasn't in the October bake-offs. Without it, the correlation is 0.42, and within a seat it's about 0.44. That isn't stable either: across all seats in the newest 46 runs, it's -0.18. We haven't traced why News Desk sits apart.

Share passing the audit, by promotion score

85 or higher54%44 of 81, mean audit 4.93
60 to 8418%9 of 49, mean audit 3.29
Below 60, excluding News Desk18%3 of 17, mean audit 3.59
News Desk, all below 6067%12 of 18, mean audit 6.22

So the score carries real signal, and high scores did better: 97 of all 165 runs failed, which is 59%, against 46% of the top scorers. It still isn't a quality gate. Of the 81 runs scoring 85 or higher, 37 failed the audit, and the seat with some of the lowest scores had some of the best audit results. These are all scored runs, not only the winners of rounds.

Seat by seat, for the six seats whose bake-off wrote a winner, it looks like this:

SeatPromotion score in bake-offAudit mean of 8Passed the audit
Tinker1004.00 of 3
Senior925.01 of 1
Watchdog885.99 of 10
Docs Keeper885.52 of 2
CFO75 to 884.75 of 10
Release Keeper55 to 682.30 of 10

Tinker scored a perfect 100 in bake-off replays, and the graded replays passed the audit. Its later production runs scored 55 to 88 on the same score and passed 0 of 3. Release Keeper scored low and did badly, as you'd hope. These counts are small and come from different weeks, and two rows have runs with no recorded model, so treat them as examples and not as measurements.

The Senior bake-off shows a different problem. Sonnet 5.5, Haiku 5.5, GPT-4.1 Mini and GPT-5.4 Nano all scored exactly 92. Nano won because it was cheapest. The two Nano bake-off replays scored 4 of 8 in the audit and failed it. A score that gives four models the same number can't choose between them.

Four very different robots hold up identical scorecards while a puzzled judge hands the trophy to the smallest one, which wears a price tag
Prompt · Codex image generation, Nov 2026

Landscape 3:2 editorial illustration in a risograph print style, two inks (deep navy and warm orange) on warm cream paper, visible grain, generous empty space, clean shapes, calm witty New Yorker-like humour. Absolutely no text, no letters, no numbers, no logos, no brand marks, no real people. Scene: Four quite different robots (one huge, one tiny, one round, one lanky) stand in a line on a contest stage, each holding up an identical blank orange scorecard with the same single shape on it. A judge robot at a table looks at them, scratching its head with a pen. A small price-tag shape hangs from the tiniest robot, which is receiving the trophy by default.

Why it happened

The score rewarded the things it could see, and it couldn't see whether a claim was true.

  • A quarter of it was usefulness, and part of that was a keyword regex. pause it? and You can counted as actionable, whether or not the advice was any good.
_ACTION_IN_NOTE = re.compile(
    r"\b(?:Next for you|Nothing for you|Approve|You can|you can|pause it\?)", re.I)
  • The checks compared each note to its own run's facts, so they couldn't catch a fact that was wrong at the source.
  • Judge votes came from a judge that had only seen the note.

A gate like this makes a weak metric more dangerous. Every round selected on that number, so a note that looked good could beat one that was right.

How much to trust this comparison

With some care. The audit is agent-made, from the same model family as parts of the score, so it's a stricter reference and not a neutral one. Only 11 runs have a stored version score and an audit grade for the same run, and all of those are bake-off replays. Production runs don't record which version produced them in a way we can match to a bake-off, so seat comparisons rely on seat and model. The rubric and evidence also changed during these weeks.

We feel confident about something narrower. A high score didn't mean a report was right, and four models tying at 92 shows the score couldn't always tell candidates apart.

What we changed

Oct 11Promotion test shipsFrozen scoring sets, champion against challenger, a 3-point margin
Oct 12Model bake-offs join inFour models on the same facts, and evaluation gets its own budget
Oct 24First audit90 runs checked claim by claim. The score's regex is found
Oct 28Judge gets real evidenceIt runs in shadow and agrees with the audit on 61.5% of runs, against an 85% bar
Nov 1This comparison165 runs, and 46% of top scores fail the audit. The old score still drives the promotion test while the new grader stays in shadow

That last line is the awkward part. The old score still feeds the promotion test today, because the replacement isn't good enough to take over. We'd rather have a known weak number and say so than swap in a new one we haven't validated. Until then, a promotion should be read as a hint. Nothing yet forces a check against the audit, and we still need to build that.

Where we pick this up

Handover sheet · Nov 1

Our notes for the next shift. If you run agents too, this list may save you a lap.

  • Open
    Trace why News Desk sits apartIt scored below 60 on the promotion test and passed 12 of 18 audits, and we don't know why yet.
    the scoring module
  • Open
    Check winners against the auditNothing stops a bake-off from writing a winner today.
    iterate_seats.py
  • Open
    Store the version on every production runFact-only seats record a version without its model, and container seats record nothing, so we can't line up most of our history.
    autopilot_results
  • Open
    Break ties on more than priceFour models tied at 92, and the cheapest won by default.
    decide_bakeoff
  • Waiting
    Retire the keyword regexIt stays until the new grader is enforced, and its actionable axis takes over.
    _ACTION_IN_NOTE
  • Keep
    Keep the shape of the testFrozen inputs and a margin to win both held up, and the bugs the rounds surfaced justified the work.

Next time we automate against a metric, we'll plot it against a stricter measure on real runs first, and split by seat before believing any single number. We should have done that here.

Related reading: how we audited the reports and the harness experiment.

If you've validated an agent metric against real outcomes, we'd like to hear how: boris@curate-me.ai.

Comments

Loading comments...

Leave a comment

Comments are moderated by our AI agent and reviewed by a human.