# What 1,000 CI Runs Taught Us About Making CI Work for Agents

> Most of our pull requests are opened by agents, and CI is their feedback loop. We pulled a month of runs, found where the minutes and the confusion went, and hill-climbed it in four measured rounds. Here's what moved and why.

- Source: https://its-boris.com/blog/ci-hill-climb-for-agents
- Author: Boris Barash
- Published: 2026-11-07
- Agent-led: yes, the work behind this post was done by agents
- AI cost: $22.63

191 of the last 200 merged pull requests on the Curate-Me platform were written by agents. For them CI isn't a dashboard someone glances at; it's the loop. Push, wait for a verdict, read why it failed, fix, push again. Every minute of waiting and every megabyte of log they have to read is a minute and a megabyte in every round of that loop.

So we treated CI as something to hill-climb. We pulled a month of runs, found where the time and the confusion went, and changed one thing at a time, keeping each change only if real runs got faster with nothing turning red. Four rounds, all merged and live the same day. This is what we found and what moved.

- Node setup, per job: 5.3 min to 2 s. A 6.4 GB cache download the runner didn't need
- Node setup share of runner time: 34% to under 1%. Of all CI job minutes, the week before vs the 9 runs after
- Reading a failed run: 2.8 MB to 7.7 KB. Raw log vs gh run view, which now lists the failing tests
- Slowest suites: about 2x faster. Mobile facade 3.7 to 1.8 min, runner-GA 4.7 to 2.4 min, first 2 runs

## How we climbed

![Flowchart. Pull 1,000 CI runs, then find where the minutes go. Then a loop: change one thing, measure on real runs, and ask whether it's faster with nothing red. Yes: keep it. No: revert it. Both go back to change one thing. Below the loop, review, then pick the next change, which also leads back to change one thing.](https://its-boris.com/images/blog/ci-hill-climb/climb-loop.jpg)

*The loop. The orange diamond is the rule every change had to pass.*

The method is the same one we used to [hill-climb a decision model](/blog/jev-hill-climbs-four-jobs), pointed at our own build:

1. **Measure before touching anything.** We pulled the last 1,000 runs of the main CI workflow with the GitHub API, including every job's and every step's start and finish time. 998 of them had full job data.
2. **Write the score down.** The headline number is *time to verdict*: from a push to the moment the PR Gate, the one check an agent waits on, says pass or fail. Beside it: how often main is green, runner-minutes per run, and how much an agent has to read to learn why a run failed.
3. **Change one area per round, and measure it on real runs.** Each round went out as its own pull request (rounds 1 and 2 shared one, touching different jobs), ran the full CI on itself, and was compared job by job against the weeks before.
4. **Keep it only if real runs get better and nothing turns red.** Main stayed green through every merge, and every deploy printed its receipt: "All verified services run the deployed commit."
5. **Review before choosing the next round.** After the first two changes we stopped and looked again, and the review changed what we worked on next.

We committed the measurement as a script, `scripts/ci/ci_stats.py`, so every round is scored the same way and anyone can rerun it.

## What a month of runs showed

The starting point was already much better than a month earlier. An October speed plan had cut the PR wait from about an hour to about 20 minutes, and main had gone from green on 17% of runs one week to 94% the week we started. The question was where the remaining 20 minutes went, and what an agent goes through when a run fails.

Four things stood out.

**Every Node job downloaded a 6.4 GB cache it already had.** `actions/setup-node` with `cache: npm` restores the npm cache from GitHub's cache service. Our runners are self-hosted, and the npm cache directory already lives on their disk, warm. The restore downloaded a 6,400,171,819-byte tarball at about 40 MB/s into it anyway: a median 5.3 minutes in each of four jobs, while the `npm ci` after it took 0.7 minutes. Over the month that was 6,325 runner-minutes, 18% of all CI time. In the last week it was 34%, and growing: on each lockfile change the action saves the whole directory again under a new key, and nothing prunes it.

**The longest job ran on almost every PR.** The Office Playwright suite, 67 browser tests that took 13.4 minutes, ran on 110 of the last 116 code PRs and was the slowest job in 84% of them. Its path filter included the whole backend and every script, but the tests run against the dashboard with a stubbed backend.

**A failed run was very hard to read.** On a recent failing PR, one frontend assertion came with 22,418 lines of log, 2.8 MB. The first useful line was line 1,167. That's hundreds of thousands of tokens for an agent before it finds the cause, or a truncated read that misses it. Jest printed every passing test, pytest printed every local variable in every failing frame, and the main backend suite stopped at its first failure, so an agent could fix only one backend test per push-and-wait round.

**Retries were silent.** The main backend suite ran up to three attempts and the runner-GA suites two. A flaky test showed up as a slow green, and nobody saw it.

## Round 1: stop downloading what's already on disk

We removed `cache: npm` from the seven `setup-node` steps that run on our self-hosted runners. Nothing else changed: the runner's own npm cache stays where it was, so `npm ci` stays fast.

*Chart on the page: Minutes per job or suite. Before: medians over Oct 26 to Nov 6. After: medians over the 9 runs since round 1 for the first four rows, and the mean of the first 2 runs since round 4 for the last two. The first four gaps are almost all the cache download; the last two are round 4.*

Node setup went from 5.3 minutes to about 2 seconds in every Node job. Over the 9 runs since the merge, the Frontend job's median went from 6.8 to 1.9 minutes, Repo Guards from 6.9 to 3.9 and Office Playwright from 13.4 to 9.2. Node setup's share of all runner time fell from 34% to under 1%, and the runners stopped writing 6.4 GB per Node job, up to four times a run, to a disk that has filled up on us before.

## Round 2: run the slow suite when it can change

We narrowed the Office Playwright filter to what the tests can actually see: the dashboard, the JavaScript packages and the exported Office contracts, which are the only way backend code reaches them. Replayed against the last 116 code PRs, the suite would run on about half of them: 50 instead of 110. Pushes to main still run it every time before a deploy, so nothing reaches production without it.

In the two weeks before, the few PR runs that skipped this suite (7 of 126) got their verdict in a median 12.8 minutes, against 23.5 for those that ran it. That's the gap round 2 aims at.

## The review: the run was queueing behind itself

After two rounds the job times had dropped a lot, but the first PRs' verdict times hadn't dropped as much. So we drew the timeline of every job in the new runs.

A single run fans out about 11 jobs at once onto 5 runners, so a run's jobs wait for each other as well as for other runs. On one backend-only PR, Backend Fast Checks waited 13.9 minutes to start. The timelines suggested the verdict now tracked the total runner-minutes in a run, about 37, more than its longest job.

Setup was only about 6 of those 37 minutes. The rest was tests, and the two biggest suites ran single-threaded. That review set the next two rounds: one for the agent reading the results, one for the minutes.

## Round 3: tell the agent what failed

Agents and people both look at the run summary first, so that's where the answer should be. We added a small pytest hook, `tests/ci_digest.py`, that runs only on GitHub Actions and only when tests fail. It writes a short table to the job summary, one row per failing test with the file, the line and the first line of the error, plus a command that reruns exactly those tests locally. It also turns each failure into an annotation, up to ten per job, so `gh run view` lists them without downloading any log. Jest and the Playwright suite got their own GitHub reporters, which do the same for frontend failures.

To prove it on real CI, we pushed a temporary commit with a failing Python test, a flaky one (it passed on its retry, as designed) and a failing Jest test. This is what `gh run view` printed, trimmed:

```text
X AssertionError: digest probe: org is over its daily quota
Backend Tests: services/backend/tests/b2b/unit/test_zz_ci_digest_probe.py#8

X expect(received).toEqual(expected) // deep equality
  -   "plan": "team",
  +   "plan": "free",
    at Object.toEqual (__tests__/zz-ci-digest-probe.test.ts:4:30)
Frontend: apps/dashboard/__tests__/zz-ci-digest-probe.test.ts#4

X PR gate failed: backend-test (failure) frontend (failure). Failing tests and
  rerun commands: gh run view 42393289519 (annotations) or the run summary page.
```

The whole `gh run view` output was 7.7 KB. The same run's `--log-failed` was 1.75 MB.

The rest of round 3, in one pull request:

- **Every failure in one run.** The backend suite no longer stops at its first failure, so one push-and-wait round shows all of them.
- **Retries rerun only what failed.** A retry now runs just the failed tests (`--lf`), in seconds instead of the whole suite. If they pass, the run says so: a "Flaky tests" warning and a line in the summary naming them, instead of a quiet green.
- **Less noise.** CI turns off pytest's local-variable dumps and Jest's list of every passing test. Both stay on for local runs, where they help.
- **The gate points at the answer.** A failed PR Gate lists the failed lanes and says where the failing tests are.

The probe run also caught something we'd missed. An existing test executes the real retry block from the workflow file against stub commands, and it still expected the old behaviour. It showed up as eight annotations, each with its file and line, and we updated it to pin the new contract: full suite first, failed tests only on retry, flakes reported.

## Round 4: give the slow suites three workers

The mobile facade suite (6,905 tests) and the runner-GA suites (7,803) ran single-threaded, because some of their tests share fixtures and check flows that must run in order. We moved both to three workers with pytest-xdist's `--dist loadfile`, which keeps each file on one worker in its original order and runs whole files side by side. The order inside a file, which is what the single thread protected, doesn't change.

Before switching, we ran both suites locally against throwaway Redis and Mongo containers: once single-threaded, then three times on three workers.

*Round 4, measured locally, then on CI*

| Suite                   | Single-threaded | 3 workers, by file | Every run           |
| ----------------------- | --------------- | ------------------ | ------------------- |
| Mobile facade, local    | 145 s           | 62 s               | 6,905 passed        |
| Runner-GA suites, local | 146 s           | 68 s               | 7,803 passed        |
| Mobile facade, on CI    | 3.7 min         | 1.7 and 1.9 min    | Green, first 2 runs |
| Runner-GA suites, on CI | 4.7 min         | 2.3 and 2.4 min    | Green, first 2 runs |

## What happened

Everything was merged and deployed on Nov 7. Since the first merge, main has been green on 5 runs out of 5, and every deploy printed its receipt.

*Before the climb and after, on real runs*

| Measure                         | Before          | After                   | Runs after                          |
| ------------------------------- | --------------- | ----------------------- | ----------------------------------- |
| Node setup per Node job         | 5.3 min         | about 2 s               | every run since                     |
| Node setup share of runner time | 34%             | under 1%                | 10 runs                             |
| Runner-minutes per run, median  | 40.7            | 37.8                    | 10 runs since round 1               |
| Mobile facade tests             | 3.7 min         | 1.7 and 1.9 min         | first 2 runs with round 4           |
| Runner-GA suites                | 4.7 min         | 2.3 and 2.4 min         | first 2 runs with round 4           |
| Main, push to deployed          | 34.0 min median | 12.2 min                | first main run with all four rounds |
| Reading a failed run            | 2.8 MB of log   | 7.7 KB from gh run view | the probe run                       |

Round 4 takes about four more runner-minutes out of every run on top of the 37.8, and the first main run with every round in it went from push to deployed production in 12.2 minutes, against a median of 34.0 over the two weeks before.

For an agent, the loop now looks different at both ends. It waits on fewer minutes of setup and fewer single-threaded suites. When a run fails, one command, `gh run view`, tells it which tests failed, on which line, with what error and how to rerun them, and every failure in the run is there at once. A retry that rescues a flaky test now says so.

## What we kept

| Change                                                                         | Where          | Status                                  |
| ------------------------------------------------------------------------------ | -------------- | --------------------------------------- |
| No npm cache restore on self-hosted runners                                    | platform #4081 | Live, Nov 7                             |
| Office Playwright only when Office can change; always on main                  | platform #4081 | Live, Nov 7                             |
| The scorecard script, scripts/ci/ci\_stats.py                                  | platform #4081 | Live, Nov 7                             |
| pytest failure digest and annotations; Jest and Playwright failure annotations | platform #4083 | Live, Nov 7                             |
| Every failure in one run; retries rerun only failed tests; flakes reported     | platform #4083 | Live, Nov 7                             |
| Mobile facade and runner-GA suites on three workers, by file                   | platform #4084 | Live, Nov 7 (deploy verified 17:58 UTC) |

## Lessons we're keeping

1. **Measure the agent's loop, not the jobs.** Time to verdict and bytes to the first cause are what an agent feels. Job times are how you find out why.
2. **Read the logs of your fast steps too.** The biggest single waste was a setup step everyone assumed was a cache hit. It was a hit: of a 6.4 GB download.
3. **Defaults built for hosted runners can cost you on your own.** The npm cache action is right on a fresh machine and wrong on one that keeps its disk.
4. **Draw the timeline after every change.** Once the jobs got shorter, the run was queueing behind itself, and the next round had to cut total minutes, not the longest job.
5. **Write the failure for the reader who'll fix it.** A short table of failing tests with a rerun command is worth more to an agent than any amount of log. Prove it with a failing probe on real CI before you trust it.
6. **Make retries say what they did.** A retry that passes should leave a mark, or the flake stays forever.
7. **Keep each round small enough to read.** Each round touched its own jobs, so its effect shows up in its own rows of the table, and a review between rounds tells you where to look next.

## Where we pick this up

Our notes for the next round, kept short.

- Shipped: Rungs 1 and 2. No npm cache restore on self-hosted runners. Office Playwright runs when Office can change, and always on main.
- Shipped: Failure digest and visible flakes. pytest: a per-test table and rerun command in the job summary. pytest, Jest and Playwright: failure annotations for gh run view. Every failure in one run, retries on failed tests only.
- Shipped: Parallel slow suites. Mobile facade and runner-GA on three workers by file.
- Keep: Score every round the same way. python3 scripts/ci/ci\_stats.py --since \<merge time>. Time to verdict p50 and p90, green rates, runner-minutes per run, critical-path job.
- Next: Re-measure over a full working week. Saturday traffic is light. Rerun the scorecard after a week of normal PRs, and count the flaky-test warnings the retries now raise.
- Next: Keep cutting runner-minutes per run. The Office browser suite runs its 67 tests on one worker, and the Python lanes each install and start their own Redis and Mongo.

## What to know before relying on this

Job times before the climb are medians over Oct 26 to Nov 6. Job times after are from the runs since each merge, all on Saturday, Nov 7: 10 runs for rounds 1 to 3 and the first 2 for round 4, so the after side has few runs and light traffic. Single-step numbers like the cache download are mechanical and repeat run to run; the 12.2-minute main run is one run. We haven't quoted a new PR time to verdict yet: most PRs since the merges were our own CI changes, and a change to the workflow file runs every job by design. We'll rerun the scorecard after a working week. The Office filter's 110-to-50 figure is a replay of the last 116 PRs' changed files against the new filter, not a count of new PRs; a second replay by the fact-check agent got 114 to 53. The round 4 local timings were three runs on three workers each, plus one single-threaded run, against throwaway containers.

Everything here was run by Claude Code agents: the analysis, the four pull requests, the probe and the measurements. The raw numbers, the method and every round are in the platform repository at `docs/plans/CI_HILL_CLIMB_RESEARCH_2026_11_07.md`.

## AI cost of this post

- Claude: research, four rounds, measurement and this note: $20.00. API-equivalent at Opus rates, estimated from this session's tokens
- Claude: fresh fact-check: $2.00. API-equivalent, from the fact-check agent's 147k tokens, estimated
- Codex: cover and flowchart, 3 images, none thrown away: $0.63. estimate, about $0.21 per image
- CI runs for the four pull requests: $0.00. our own self-hosted runners, no metered minutes

Claude and Codex ran on flat subscriptions, so their lines show the equivalent at API rates.