What it costs when machines write the code
Last month, 66,931 lines were added to the Promise compiler across 249 resolved tasks — planned, implemented, reviewed, tested and landed.
I wrote none of them.
At API list prices that work would have cost $3,078. I ran it on a Claude Max subscription — $200 a month, flat.
This is BASE — bounded-autonomy software engineering — with no human in the resolution loop. Humans own the intent and a mechanical definition of quality; the system owns the implementation. Tasks are planned, implemented, reviewed, brought up to test coverage and landed across a fleet of 29 arenas spread over macOS, Linux (amd64 and arm64) and Windows, with 22 of them active in the period below.
Worth being precise about "no human in the loop", because it doesn't mean I walked away. My own estimate is about an hour a day — deciding design questions, sharpening intent, unblocking things that got stuck. What that hour never includes is writing code or reviewing a diff. That is the boundary the whole system is built around: not absent, just not in the resolution loop.
The project being built is Promise, a programming language — but almost nothing here is about language design. It is what it costs to build software this way: 1,371 tasks since May, 2,043 commits, and 18,869 tests — which, run across the platforms, comes to 105,791 executions that must all be green before anything lands.
Below is what that costs and how fast it moves, including the parts that don't flatter it.
Where the numbers come from
The project's task tracker records what each resolution consumed. Every figure here covers 2026-05 onward — before that, too much was run one-off and untracked to compare against, so including it would only make the numbers look better and mean less.
That leaves 1,371 tasks, of which 863 (63%) carry a recorded cost. Every total below is therefore a floor, not a full accounting.
One task, $14.67 — near enough the median. Seven artifacts, an inspection verdict with quality and completeness, and a follow-up the system raised on its own. This record, times 1,371, is where every number below comes from.
One thing to be clear about up front, because it changes how you should read all of this. The recorded cost is what the work would have cost at API list prices. It all ran on a Claude Max subscription, at a flat monthly fee. So these are not invoices. They're the meter reading on a plan that doesn't charge by the token.
The distribution
| p25 | $6.74 |
| median | $12.62 |
| p75 | $20.30 |
| p90 | $31.79 |
| max | $190.92 |
The mean is $16.06, and quoting it would be misleading — the top 10% of tasks account for 31% of all spend. Most work is cheap. A few things are genuinely brutal.
The $190.92 outlier was a generator holding a heap-allocated default argument that its structural interface adapter had already freed — the parameter default is released at the end of the statement, but the coroutine frame reads it lazily, later. Ownership bugs that cross an abstraction boundary are the expensive ones, every time. That's the class of bug that costs a person a week and a half of staring at a debugger.
If it sounds familiar, it's the one I described a week ago at the end of 95% of the test suite was leaking — "a generator coroutine reading a default argument its structural interface adapter had already freed." That post is what the bug looked like. This is what it cost.
You don't have to take my word for any of it. That one is issue #47, and the commit that resolved it touches 10 files and adds 2,179 lines. That is what $190.92 bought.
At the other end, the cheapest resolution in the whole set was $0.20 — a Windows parity bug where the blob store failed to reject a non-directory path. That one is issue #48, and its fix is one file, fourteen lines.
Those two are worth holding side by side. A 950× spread in price bought a 155× spread in diff — which is roughly what you'd hope for, and not at all what you'd get from a system that spent proportionally to how hard it was told something was.
How long a task actually takes
There are two answers, and the gap between them is the interesting part.
Wall-clock, from filed to resolved: median 15.7 hours, 56% inside a day, and a tail running to 27 days at p90.
Actual working time: median 44 minutes.
| p25 | 19.7 min |
| median | 44.0 min |
| p75 | 64.2 min |
| p90 | 82.5 min |
| under an hour | 69% |
Almost all the elapsed time is queue, and most of that is the throttle described further down: with one task allowed to run at a time, work stacks up behind work.
On top of that, each platform runs a stress pass once a day. A stress run takes every Promise test — 10,494 of them — and loops it a hundred times, running concurrently at one test per available CPU core, watched for failures, hangs and memory problems. That is over a million test executions in a single pass, per platform, every day. Running things once tells you the code works. Running them a hundred times under full load is how the rare ordering bug and the leak that only appears under pressure surface at all.
It saturates that machine while it runs, so nothing else there can validate:
| stress pass | tests/second | |
|---|---|---|
| Linux | 38 min | 460 |
| macOS | 77 min | 227 |
| Windows | 216 min | 81 |
Linux clears 460 test executions a second, sustained, for thirty-eight minutes. That's only possible because the toolchain is built for it: the tests are compiled once and the binaries re-run, so a stress pass is a hundred executions rather than a hundred build-and-test cycles. Done the naive way it would take far longer than a machine has in a day, and the gate simply wouldn't exist.
The throughput also makes the scheduling behaviour arithmetic rather than mysterious: Linux is twice macOS and nearly six times Windows, so an orchestrator taking whatever frees up first drifts toward Linux without anyone designing that.
A task finishing inside that window sits finished-but-unverified until the run clears.
Writing the code is a third of the work
Every step of a task resolution is metered separately, in both dollars and minutes. I expected this to come out roughly where it did — it's the assumption the system is built on — but assumptions are cheap and this one is now measured:
In flow order, as a resolution actually runs:
| step | median cost | share of spend | share of time | retries |
|---|---|---|---|---|
| create plan | $2.15 | 18% | 17% | 1.10 |
| implement | $3.01 | 34% | 38% | 1.17 |
| review and fix issues | $2.24 | 17% | 12% | 1.03 |
| fill coverage gaps | $1.80 | 13% | 19% | 1.07 |
| commit and push to origin | $1.73 | 8% | 14% | 1.47 |
| summarize resolution | $0.41 | 2% | — | 1.00 |
| inspect | $1.48 | 8% | — | 1.01 |
Writing the code is about a third — 34% of the money, 38% of the time. The other two thirds is planning, reviewing, closing test-coverage gaps, inspecting and landing it. Cost and time agree closely enough that neither is an artefact of how one of them is measured.
A few things fall out of that table.
Review plus coverage plus inspection is 38% of spend — as much as writing the code. That is the price of keeping a person out of the loop, and it is not overhead to optimise away; it's most of what separates a system you can walk away from and one that generates confident nonsense — which is the argument I made qualitatively in The anatomy of the process, now with a price on it.
Commit and push retries more than anything else — 1.47 invocations against 1.0–1.2 everywhere else. Landing means rebasing onto a branch that two dozen other arenas are also pushing to, and losing that race is the single most repeated action in the system.
Plans get created more often than implementations happen — 693 against 570. Plans get made, thrown away and remade, and at $2.15 a go that's the cheapest place in the pipeline to change your mind.
Why a task is a big thing
"249 tasks a month" sounds like ticket-shuffling until you look at what a task is. Matching tracked tasks to their commits in the public repo, the median one adds 278 lines across 4 files:
| lines added | files | |
|---|---|---|
| p25 | 153 | 2 |
| median | 278 | 4 |
| p75 | 471 | 6 |
| p90 | 872 | 10 |
That size is deliberate, and the step table explains why. Commit and push, inspection and summarising come to $3.62 a task — 29% of a median task — and that charge barely moves with how large the task is.
Validation works the same way and is the bigger part of it. A cold run of the verification suite takes something like ten minutes on macOS or Linux and twice that on Windows, and a task rarely gets away with running it once. None of that scales down with the size of the change: verifying a 100-line diff costs about what verifying a 400-line one does.
So the system is told to prefer fewer, larger, more complex tasks. Split a 400-line change into four 100-line ones and you pay the fixed charge four times and buy nothing with it.
I can put a number on the money but not on the time. The orchestrator logs its own verification steps, but every suite run an agent performs inside implementation or coverage is folded into that step's duration and can't be separated out. So the $3.62 is measured; the ten-to-twenty minutes is my figure from watching it run, and I'd treat it as an estimate.
The number I actually wanted
Those tasks consumed, at API list prices:
| May | $3,806 |
| June | $3,910 |
| July | $3,078 |
| August | $3,061 (month partial, 34% priced — a floor) |
Roughly $3,000–3,900 a month, steadily, for four months.
The subscription is $200.
That is a price ratio, not a productivity claim. Same tokens, different meter. It says nothing about whether the work is fast or the code is good. What it does decide is whether one person can keep a fleet running against a real codebase — and on that question it runs about 15× in favour of the subscription.
And 15× is a floor, for a reason worth spelling out. That subscription isn't dedicated to this project. The same plan concurrently covers the Reactor normative documents, the internal orchestrator implementation, and everything else I do — none of which appears anywhere in these numbers. So the $200 bought all of the above and the $3,000-odd of work measured here, while this project competed with that other work for the same rate limits. A dedicated subscription would show a larger ratio, not a smaller one.
July is the month to trust most: 98% of its tasks carry a cost, 249 resolved, $3,077.90.
The constraint is tokens, not machines
Here is the part that decides how fast any of this goes.
The system, mid-afternoon on a working day. Twenty arenas.
Nothing executing. A task parked and tagged rate-limited.
It resumes at 04:59, when the window rolls.
That screenshot is the argument, so it's worth reading the header slowly.
20 arenas · max 1. There are twenty arenas available and
the orchestrator is allowing exactly one task to run at a time. Not
because nineteen are broken — because running more would blow the weekly
quota.
throughput 7 / 50. Seven tasks against a target of
fifty. That is the system grading its own pace, and it matches what the
task store says independently: 8.7 tasks a day on average, against
roughly 33 that a single lane could clear at 44 minutes each.
quota pace: 7d utilization 99.0% ≥ 98.0% pace target.
This is the mechanism. The orchestrator is not starved at random — it is
deliberately throttling itself to land just inside the
seven-day quota, and it is doing that accurately: 99.0% against a 98.0%
target.
Meanwhile 339 tasks sit open in the queue, about six weeks of backlog at the current rate.
So the arenas are not busy. They are waiting, on purpose, because the budget only stretches to one at a time.
The same moment from the fleet's side. Two arenas leased, the rest idle.
That is the whole reason the money matters, and it's why the
arithmetic in the sponsorship line is literal rather than promotional:
the machines are already bought and already idle. More budget raises
max 1, and nothing else has to change.
It also explains a number from earlier that looks bad in isolation. Wall-clock time to resolve has a median of 15.7 hours against 44 minutes of work — that gap is this constraint, seen from the other end.
Where this is weak
If you're going to poke holes, these are the places I'd start.
Coverage is 63%. Every total is a floor. August is only 34% priced, so its $3,061 is the softest figure on the page. Separately, and in the same direction: the subscription also ran work outside this project, so it bought more than these numbers show.
The per-step figures don't sum to the per-task ones. Step records account for $10,079 of the $13,856 total. Some spend isn't attributed to a named step, so read the step table as shares, which are sound, rather than as a complete ledger.
The timing sample is thinner than the cost sample. 255 tasks have parseable step durations against 863 with a recorded cost, so treat the 44-minute median as the weaker of the two figures.
Platform makes no difference I can demonstrate. I looked, because it's the obvious question with a fleet spanning three operating systems. Only 260 tasks can be attributed to a specific arena; on those, macOS and Linux medians sit within about 15% of each other and Windows has four priced tasks, which is nothing.
In fact I can't even count the work per platform. Only 19% of tasks can be tied to a specific arena from the logs at all, and Windows arenas hardly appear in that sample — which tells you about how the orchestrator writes its event log, not about what ran where.
What I can describe is the mechanism, from running the thing. The Windows machine is slower, and it compounds. A cold verification run takes about twice as long there. The daily stress pass takes 216 minutes against Linux's 38 — nearly six times — and nothing else on that machine can validate for the duration. Its arenas stay occupied longer and free up less often, so the scheduler — which simply takes whatever is available — reaches for macOS and Linux more. The system routes around its slowest platform without being told to.
That's sensible behaviour, and it's the part worth knowing. It also means any platform comparison here would carry a scheduling bias on top of the task mix — Windows-specific bugs skew shallow, the deep ownership bugs land wherever an arena is free — so there isn't one.
14% of tasks closed without a code change — 187 of 1,371. They were resolved, just not by fixing anything: 121 duplicates, 42 that couldn't be reproduced, 20 declined, and 4 that turned out to be working as intended. Reaching those verdicts still costs money, and I'm not netting it out of the per-task figures.
The gates, as they actually looked while I was writing this: 20 of them, 85% passing, two red. That is the floor quality is held to — not a claim that the code is good.
Cost says nothing about quality. A cheap task is not a good one. What Promise has instead is a set of mechanical gates — every commit must keep all 105,791 test executions green across Linux, macOS, Windows and WASM, and the leak count can only go down. That's a floor, not a verdict. Whether agents can build genuinely good software this way is still the open question, and the reason the Zoo publishes the failures too — see an agent writing a language it was never trained on for what that looks like when it works.
Why publish it
Because almost nobody does. The conversation about AI engineering is saturated with qualitative takes and nearly empty of unit economics, and you can't reason about a workflow whose cost you've never measured.
If you're running agents against a real codebase, I'd like your number — what a task costs you, and whether you measure it at all.
Promise itself is early and not production-ready: the compiler still crashes and the standard library is thin. Worth saying plainly before anyone decides to build something on it. The repo is here if you want to look anyway.
Comments