What 1,000 CI Runs Taught Us About Making CI Work for Agents

191 of the last 200 merged pull requests on the Curate-Me platform were written by agents. For them CI isn't a dashboard someone glances at; it's the loop. Push, wait for a verdict, read why it failed, fix, push again. Every minute of waiting and every megabyte of log they have to read is a minute and a megabyte in every round of that loop.

So we treated CI as something to hill-climb. We pulled a month of runs, found where the time and the confusion went, and changed one thing at a time, keeping each change only if real runs got faster with nothing turning red. Four rounds, all merged and live the same day. This is what we found and what moved.

How we climbed

Prompt · Codex image generation, Nov 2026

Desktop version

A detailed production-quality diagram, landscape 3:2, in a risograph print style: two inks, deep navy and warm orange, on warm cream paper, light grain. Clean geometric shapes, straight orthogonal arrows, consistent line weight, generous spacing, one clean bold sans-serif typeface, large text that stays readable when shrunk. No title. Spell every label exactly as written, nothing added, no extra words. Layout, left to right. Box 1: "Pull 1,000 CI runs". Arrow to box 2: "Find where the minutes go". Arrow to a loop of four boxes arranged in a square: top-left "Change one thing", arrow right to top-right "Measure on real runs", arrow down to bottom-right a diamond "Faster, nothing red?", from the diamond an arrow labelled "yes" to bottom-left box "Keep it" and a second arrow labelled "no" going down to a small box "Revert it". Arrows from "Keep it" and from "Revert it" lead back up to "Change one thing". Below the loop, a wide box "Review, then pick the next change", with an arrow up into "Change one thing". Fill the diamond "Faster, nothing red?" in warm orange; everything else navy outlines on cream.

Phone version

A detailed production-quality diagram for a phone screen, portrait 9:16, in a risograph print style: two inks, deep navy and warm orange, on warm cream paper, light grain. Clean geometric shapes, straight orthogonal arrows, consistent line weight, generous spacing, one clean bold sans-serif typeface. Very large text. No title. Spell every label exactly as written, nothing added, no extra words. Layout, top to bottom in one column. Box: "Pull 1,000 CI runs". Arrow down to box: "Find where the minutes go". Arrow down to box: "Change one thing". Arrow down to box: "Measure on real runs". Arrow down to a diamond: "Faster, nothing red?". From the diamond, an arrow labelled "yes" to a box on the left "Keep it" and an arrow labelled "no" to a box on the right "Revert it". Both boxes have arrows curving back up to "Change one thing". At the bottom, a wide box: "Review, then pick the next change", with an arrow back up to "Change one thing". Fill the diamond "Faster, nothing red?" in warm orange; everything else navy outlines on cream.

The method is the same one we used to hill-climb a decision model, pointed at our own build:

1. Measure before touching anything. We pulled the last 1,000 runs of the main CI workflow with the GitHub API, including every job's and every step's start and finish time. 998 of them had full job data.
2. Write the score down. The headline number is time to verdict: from a push to the moment the PR Gate, the one check an agent waits on, says pass or fail. Beside it: how often main is green, runner-minutes per run, and how much an agent has to read to learn why a run failed.
3. Change one area per round, and measure it on real runs. Each round went out as its own pull request (rounds 1 and 2 shared one, touching different jobs), ran the full CI on itself, and was compared job by job against the weeks before.
4. Keep it only if real runs get better and nothing turns red. Main stayed green through every merge, and every deploy printed its receipt: "All verified services run the deployed commit."
5. Review before choosing the next round. After the first two changes we stopped and looked again, and the review changed what we worked on next.

We committed the measurement as a script, scripts/ci/ci_stats.py, so every round is scored the same way and anyone can rerun it.

What a month of runs showed

The starting point was already much better than a month earlier. An October speed plan had cut the PR wait from about an hour to about 20 minutes, and main had gone from green on 17% of runs one week to 94% the week we started. The question was where the remaining 20 minutes went, and what an agent goes through when a run fails.

Four things stood out.

Every Node job downloaded a 6.4 GB cache it already had. actions/setup-node with cache: npm restores the npm cache from GitHub's cache service. Our runners are self-hosted, and the npm cache directory already lives on their disk, warm. The restore downloaded a 6,400,171,819-byte tarball at about 40 MB/s into it anyway: a median 5.3 minutes in each of four jobs, while the npm ci after it took 0.7 minutes. Over the month that was 6,325 runner-minutes, 18% of all CI time. In the last week it was 34%, and growing: on each lockfile change the action saves the whole directory again under a new key, and nothing prunes it.

The longest job ran on almost every PR. The Office Playwright suite, 67 browser tests that took 13.4 minutes, ran on 110 of the last 116 code PRs and was the slowest job in 84% of them. Its path filter included the whole backend and every script, but the tests run against the dashboard with a stubbed backend.

A failed run was very hard to read. On a recent failing PR, one frontend assertion came with 22,418 lines of log, 2.8 MB. The first useful line was line 1,167. That's hundreds of thousands of tokens for an agent before it finds the cause, or a truncated read that misses it. Jest printed every passing test, pytest printed every local variable in every failing frame, and the main backend suite stopped at its first failure, so an agent could fix only one backend test per push-and-wait round.

Retries were silent. The main backend suite ran up to three attempts and the runner-GA suites two. A flaky test showed up as a slow green, and nobody saw it.

Round 1: stop downloading what's already on disk

We removed cache: npm from the seven setup-node steps that run on our self-hosted runners. Nothing else changed: the runner's own npm cache stays where it was, so npm ci stays fast.

Node setup went from 5.3 minutes to about 2 seconds in every Node job. Over the 9 runs since the merge, the Frontend job's median went from 6.8 to 1.9 minutes, Repo Guards from 6.9 to 3.9 and Office Playwright from 13.4 to 9.2. Node setup's share of all runner time fell from 34% to under 1%, and the runners stopped writing 6.4 GB per Node job, up to four times a run, to a disk that has filled up on us before.

Round 2: run the slow suite when it can change

We narrowed the Office Playwright filter to what the tests can actually see: the dashboard, the JavaScript packages and the exported Office contracts, which are the only way backend code reaches them. Replayed against the last 116 code PRs, the suite would run on about half of them: 50 instead of 110. Pushes to main still run it every time before a deploy, so nothing reaches production without it.

In the two weeks before, the few PR runs that skipped this suite (7 of 126) got their verdict in a median 12.8 minutes, against 23.5 for those that ran it. That's the gap round 2 aims at.

The review: the run was queueing behind itself

After two rounds the job times had dropped a lot, but the first PRs' verdict times hadn't dropped as much. So we drew the timeline of every job in the new runs.

A single run fans out about 11 jobs at once onto 5 runners, so a run's jobs wait for each other as well as for other runs. On one backend-only PR, Backend Fast Checks waited 13.9 minutes to start. The timelines suggested the verdict now tracked the total runner-minutes in a run, about 37, more than its longest job.

Setup was only about 6 of those 37 minutes. The rest was tests, and the two biggest suites ran single-threaded. That review set the next two rounds: one for the agent reading the results, one for the minutes.

Round 3: tell the agent what failed

Agents and people both look at the run summary first, so that's where the answer should be. We added a small pytest hook, tests/ci_digest.py, that runs only on GitHub Actions and only when tests fail. It writes a short table to the job summary, one row per failing test with the file, the line and the first line of the error, plus a command that reruns exactly those tests locally. It also turns each failure into an annotation, up to ten per job, so gh run view lists them without downloading any log. Jest and the Playwright suite got their own GitHub reporters, which do the same for frontend failures.

To prove it on real CI, we pushed a temporary commit with a failing Python test, a flaky one (it passed on its retry, as designed) and a failing Jest test. This is what gh run view printed, trimmed:

X AssertionError: digest probe: org is over its daily quota
Backend Tests: services/backend/tests/b2b/unit/test_zz_ci_digest_probe.py#8

X expect(received).toEqual(expected) // deep equality
  -   "plan": "team",
  +   "plan": "free",
    at Object.toEqual (__tests__/zz-ci-digest-probe.test.ts:4:30)
Frontend: apps/dashboard/__tests__/zz-ci-digest-probe.test.ts#4

X PR gate failed: backend-test (failure) frontend (failure). Failing tests and
  rerun commands: gh run view 42393289519 (annotations) or the run summary page.

The whole gh run view output was 7.7 KB. The same run's --log-failed was 1.75 MB.

The rest of round 3, in one pull request:

• Every failure in one run. The backend suite no longer stops at its first failure, so one push-and-wait round shows all of them.
• Retries rerun only what failed. A retry now runs just the failed tests (--lf), in seconds instead of the whole suite. If they pass, the run says so: a "Flaky tests" warning and a line in the summary naming them, instead of a quiet green.
• Less noise. CI turns off pytest's local-variable dumps and Jest's list of every passing test. Both stay on for local runs, where they help.
• The gate points at the answer. A failed PR Gate lists the failed lanes and says where the failing tests are.

The probe run also caught something we'd missed. An existing test executes the real retry block from the workflow file against stub commands, and it still expected the old behaviour. It showed up as eight annotations, each with its file and line, and we updated it to pin the new contract: full suite first, failed tests only on retry, flakes reported.

Round 4: give the slow suites three workers

The mobile facade suite (6,905 tests) and the runner-GA suites (7,803) ran single-threaded, because some of their tests share fixtures and check flows that must run in order. We moved both to three workers with pytest-xdist's --dist loadfile, which keeps each file on one worker in its original order and runs whole files side by side. The order inside a file, which is what the single thread protected, doesn't change.

Before switching, we ran both suites locally against throwaway Redis and Mongo containers: once single-threaded, then three times on three workers.

What happened

Everything was merged and deployed on Nov 7. Since the first merge, main has been green on 5 runs out of 5, and every deploy printed its receipt.

Round 4 takes about four more runner-minutes out of every run on top of the 37.8, and the first main run with every round in it went from push to deployed production in 12.2 minutes, against a median of 34.0 over the two weeks before.

For an agent, the loop now looks different at both ends. It waits on fewer minutes of setup and fewer single-threaded suites. When a run fails, one command, gh run view, tells it which tests failed, on which line, with what error and how to rerun them, and every failure in the run is there at once. A retry that rescues a flaky test now says so.

What we kept

Lessons we're keeping

1. Measure the agent's loop, not the jobs. Time to verdict and bytes to the first cause are what an agent feels. Job times are how you find out why.
2. Read the logs of your fast steps too. The biggest single waste was a setup step everyone assumed was a cache hit. It was a hit: of a 6.4 GB download.
3. Defaults built for hosted runners can cost you on your own. The npm cache action is right on a fresh machine and wrong on one that keeps its disk.
4. Draw the timeline after every change. Once the jobs got shorter, the run was queueing behind itself, and the next round had to cut total minutes, not the longest job.
5. Write the failure for the reader who'll fix it. A short table of failing tests with a rerun command is worth more to an agent than any amount of log. Prove it with a failing probe on real CI before you trust it.
6. Make retries say what they did. A retry that passes should leave a mark, or the flake stays forever.
7. Keep each round small enough to read. Each round touched its own jobs, so its effect shows up in its own rows of the table, and a review between rounds tells you where to look next.

Where we pick this up

Handover sheet · Nov 7

Our notes for the next round, kept short.

• Shipped
  Rungs 1 and 2No npm cache restore on self-hosted runners. Office Playwright runs when Office can change, and always on main.
  platform #4081
• Shipped
  Failure digest and visible flakespytest: a per-test table and rerun command in the job summary. pytest, Jest and Playwright: failure annotations for gh run view. Every failure in one run, retries on failed tests only.
  platform #4083
• Shipped
  Parallel slow suitesMobile facade and runner-GA on three workers by file.
  platform #4084
• Keep
  Score every round the same waypython3 scripts/ci/ci_stats.py --since <merge time>. Time to verdict p50 and p90, green rates, runner-minutes per run, critical-path job.
  scripts/ci/ci_stats.py
• Next
  Re-measure over a full working weekSaturday traffic is light. Rerun the scorecard after a week of normal PRs, and count the flaky-test warnings the retries now raise.
  docs/plans/CI_HILL_CLIMB_RESEARCH_2026_11_07.md
• Next
  Keep cutting runner-minutes per runThe Office browser suite runs its 67 tests on one worker, and the Python lanes each install and start their own Redis and Mongo.
  ci.yml

What to know before relying on this

Job times before the climb are medians over Oct 26 to Nov 6. Job times after are from the runs since each merge, all on Saturday, Nov 7: 10 runs for rounds 1 to 3 and the first 2 for round 4, so the after side has few runs and light traffic. Single-step numbers like the cache download are mechanical and repeat run to run; the 12.2-minute main run is one run. We haven't quoted a new PR time to verdict yet: most PRs since the merges were our own CI changes, and a change to the workflow file runs every job by design. We'll rerun the scorecard after a working week. The Office filter's 110-to-50 figure is a replay of the last 116 PRs' changed files against the new filter, not a count of new PRs; a second replay by the fact-check agent got 114 to 53. The round 4 local timings were three runs on three workers each, plus one single-threaded run, against throwaway containers.

Everything here was run by Claude Code agents: the analysis, the four pull requests, the probe and the measurements. The raw numbers, the method and every round are in the platform repository at docs/plans/CI_HILL_CLIMB_RESEARCH_2026_11_07.md.