Agent Harness vs a Single LLM Call: Why Our $0.875 Job Dropped to $0.0016

News Desk is one of our agents. Each day it reads the AI news and writes a short edition for the team. For a while it ran inside a full coding-agent harness, at about $0.875 a run and more than seven minutes. Then we moved it to a single model call, and it dropped to about $0.0016 and 37 seconds.

That's a gap of roughly 500 times, for a report our audit scored about the same. The $0.0016 is the run itself and doesn't include review calls. The obvious headline was "tune your harness before you touch your model".

We tried to prove it, and the headline fell apart.

What the first comparison actually measured

Our numbers came from production, and they looked clean:

Those runs differ in almost every way that matters. The dates, the model and the code around the job all changed together. The old version also did more: its curator fetched the news itself, reading whole pages along the way. The new version receives facts that plain code has already collected.

So the 500x gap measures a different architecture. It doesn't tell you what a harness costs.

The controlled replay

We took five real editions from October, kept the facts and the prompt, used the same model throughout, and varied only the harness in arms A and B. Thinking also differed (see the flaws below), and arm C changed the prompt.

Separate Claude agents scored each edition without knowing which arm wrote it. Each arm wrote each edition more than once, for 40 graded runs in all.

Arms A and B each took one turn, and arm C averaged five. The harness itself added about 1.9 times the cost and doubled the time. Split by thinking, the overhead runs from about 1.6x to 3.2x. The difference in score between A and B is within noise. That's a real overhead, but it's nowhere near 500 times.

We didn't expect arm B to ignore its tools. It had search, fetch and read available in all 15 runs, and it made zero tool calls, most likely because the prompt forbids outside facts and never asks for research. In one pilot, merely permitting browsing didn't change that. The tools just sat there.

So we told it to browse

That's arm C, which we added once we saw arm B doing nothing. It costs about 14 times the single call. It's also a different job, and you can see why when you read what it wrote.

Arm C's version is more specific. It went to the source and came back with details. Under our rubric it failed every time on the strict reading, because the rubric says to use the supplied facts. When we allowed outside facts that turned out to be true, 5 of 10 passed. Of the 38 outside claims in its top stories, 33 were confirmed, 5 couldn't be confirmed and none were false, and one further claim was misleading.

So we can't tell you browsing is bad. We can say that it costs 14 times more, that half its editions still carried a claim our recheck couldn't confirm or one marked misleading, and that you need to decide in your rubric what you're buying. Ours was written for a job that doesn't research.

Where the money went

Breaking down the path from $0.875:

1. Fetching moved to code. Plain code now collects the stories, so the model never reads whole web pages. This was almost certainly the biggest saving, though we can't isolate it exactly because the model wasn't recorded for those old runs.
2. The harness wrapper went away. The replay puts this at about 2x when nothing is browsing. Browsing takes it to about 14x, but that overlaps with the first item.
3. The model may have got cheaper. Haiku and GPT-4o Mini cost little at this size, but we didn't record the old model, so we can't size this.

This matches what others have found. Anthropic's guidance on agents recommends the simplest thing that works, and an imec study of coding agents found that harness choice mattered less than expected for resolving tasks, though the harness and model combination could double the cost per resolved task. The useful question is whether the model needs to decide what to do next. Ours didn't, because code already knew what to fetch.

Flaws in this experiment

• We couldn't fully turn thinking off. We tried four different switches on the CLI, and thinking still appeared in 10 of 15 runs of arm A and all 15 of arm B. So A and B differ in thinking too. This is the biggest flaw, and it may matter: within arm A, the 10 runs with thinking all passed, and 4 of the 5 without it did.
• It's one job with five inputs. All come from one org, and all are the kind of job that code can prepare.
• It's a local CLI. It isn't our production container. A real production call came very close to arm A with thinking off ($0.00132 against $0.00139, and 5.7 against 5.8 seconds), so the comparison is reasonable, but it isn't identical.
• Blinding was imperfect for arm C. Its extra facts are visible to a grader.
• Old grades were weaker than they looked. The audit passed 0 of the 3 Claude Code runs, but that came from one judgement applied to all three, so we left the pass column out above.

A bug this found

The experiment turned up something unrelated. All five inputs needed a second writer call. Of 32 completed gateway runs since October 14, 30 used the fallback, so we were paying about double for most editions. All 22 saved retry prompts failed on research lines: 14 had invented arXiv numbers, 5 had bare row numbers and 3 were non-paper stories. We suspect the prompt's "arXiv ids only" wording prompted the invented numbers. It costs fractions of a cent, so no alarm ever fired. We only noticed by looking at the raw calls.

So we fixed the prompt and stopped sending editions back over lines the server could drop anyway, then ran it live five times. All five passed on the first try with no fallback, at $0.0003 to $0.0008 each. Before the fix, 11 of the last 13 completed scheduled runs retried, at about $0.0019.

Those five flatter the fix. Our test marker didn't stick, so each run marked its stories as printed and the next one read a shorter list, which is why the fifth cost 62% less than the first. We removed those marks afterwards so the next real edition kept its stories.

The test also showed what the retry had been hiding. All five editions came back with 3 stories, the minimum, because the fallback had been what topped them up to 6. We now ask for the full size, and if the writer returns at least 3 usable stories, code tops up the rest with no second call. One live run of that came back with 6 stories on the first try, at $0.0008. That's one run, and the next scheduled editions are the real test.

The page that said everything had failed

While finishing this note, we saw that our public Night Owl page listed 14 failed runs in a row, the newest from October 6. The job was fine. It had completed on 13 of its last 14 days.

The page's API key belonged to an older org, and the job there stopped on October 6 while the one in the blog's main org kept running. Our own build log recorded the mismatch in early October, and nobody acted on it. The page now has its own key for the right org. It shows the real runs, schedule and model, and we removed an out-of-date line saying it ran free on a personal subscription.

What these experiments add up to

This note wraps up the experiments in this series.

We take three things from it.

Most of the savings came from asking the model to do less. Code fetches the facts, one call picks and writes, and code checks the result. That's the shape of the job that now costs a fraction of a cent.

A score means little until someone checks it against the records. The eval and the ratchet both looked better than they were, and Jev's confidence scores didn't hold up. We still have no automated grader that clears 85%, so an agent-led audit against production records stays our reference, and the judge stays in shadow.

Several of our bugs looked fine on the surface. An error count that could only read 0. Plan refusals reported as rate limits. A retry on nearly every run that no alarm caught. A confidence score read as a probability. None of them threw an error. Most surfaced when we read raw records, and the stale page surfaced only because someone looked at it.

Where we pick this up

Handover sheet · Nov 1

Our notes for the next shift. If you run agents too, this list may save you a lap.

• Shipped
  Fix the News Desk writer promptFive of five live runs passed first time. Lines the server can safely drop no longer force a retry.
  news_digest.py
• Shipped
  Point the Night Owl page at the right orgIt read a retired org for weeks. It now has its own key.
  /demos/cron
• Open
  Watch the next editionsOne test run produced 6 stories with no second call. The scheduled editions have to confirm it.
  autopilot_results
• Open
  Keep test runs from counting as editionsMetadata sent to the run route didn't reach the stored result, so ours marked stories as printed.
  autopilot/core.py
• Open
  Decide what the rubric wants from researchBrowsing added specifics, and our rubric marked it down. One of those has to change, depending on the job.
  News Desk objective
• Open
  Re-run with thinking under controlFour switches failed to turn it off, and it may matter more than the harness.
  run_experiment.py
• Open
  Test a job that needs to choose its next stepThis job has its facts fetched by code. A job that has to look at results and decide may need the harness.
  harness.trial
• Open
  Record the model on every runWe can't fully explain our own savings because the old runs have none.
  autopilot_results

Before we pay for a loop now, we ask whether the model needs to choose its next step, and we check whether our agents use their tools. If code knows what to fetch, it fetches first and we send one call.

The grades here use the same 8-point rubric as our agent-led audit, checked against the supplied facts instead of production records.

If you've measured harness overhead on a different kind of job, we'd love to compare: boris@curate-me.ai.