What an AI agent really costs: the accepted-task method
A 2,000-minute CI allowance hit 90% in three and a half days. The fix is not a cheaper model but a different denominator: cost per accepted task, checked by a script.
🎯 For Teams paying for AI agents who suspect the bill and the value have drifted apart.
- Replace token or subscription counters with cost per accepted task
- Write the acceptance rule and the death criterion before the first run
- Report subscription quota and human minutes instead of calling a paid plan free
- Recognise five situations where the method produces precise numbers with no meaning
A 2,000-minute monthly CI allowance was 90% gone after three and a half days, and nobody could say what the work that burned it had produced. That is the whole problem in one sentence: we could count the minutes, the tokens and the subscription, but we could not count accepted work. This page is the method we now use to count it, written after measuring our own case on 4 November 2026.
The short answer: measure the cost of one accepted task — money, human minutes, rework and overhead divided by the number of deliverables that passed a machine check on the first attempt. Counters such as tokens, requests, "credits", CI minutes or a flat monthly price are all inputs to that number, never the number itself.
Why the obvious counters mislead
A subscription makes every single run look free, because the payment already happened last month. A metered API makes every single run look expensive, because each call has a price attached. Both views are wrong in the same direction: they price the input and stay silent about the result.
Our own evidence, measured on 4 November 2026 from repository run metadata (not from a dashboard screenshot):
| Observation | Value |
|---|---|
| Monthly included CI minutes | 2,000 |
| Consumed when the warning arrived | 1,812 (90%) |
| Days used of the 30-day cycle | 3.5 |
| Runs of one workflow in two days | 109 |
| Average duration of that workflow | 14.1 minutes |
The expensive part was not the model. It was 109 pushes in two days, each starting a 14-minute job that installed dependencies and ran a full test suite. Two of those runs per push, because the same commit triggered both a branch push and a pull-request event, and nothing cancelled the older one. The agent wave produced the pushes; the bill arrived in a currency nobody was watching.
The method
- Freeze the task before you dispatch it. Write down the input, the expected output, the acceptance rule, the attempt limit and the date. Everything that judges the outcome is written before the outcome exists.
- Accept with a script, not with an opinion. For a draft, the check is that every claim matches a pre-verified fact pack and every link returns 200. For extraction, it is an exact match against a frozen schema. For code, it is the named test file plus the linter. A model may comment on quality, but it does not decide acceptance.
- Record one row per attempt. Task, class, variant, accepted, API spend, human minutes, elapsed seconds, overhead, evidence reference, timestamp, and the billing kind (
subscriptionormetered). A subscription row carriesapi_usd = 0.0and the consumed quota units — writing "free" is a defect, not a shortcut. - Count only accepted work. A rejected attempt is a cost with no deliverable: it stays in the spend column and never enters the denominator of finished tasks.
- Publish at least 30 matched tasks. Thirty is a minimum sample gate, not statistical significance. Ten per task class, so one class failing does not get reported as the general verdict.
- Announce the death criterion first. Ours: if fewer than 80% of tasks have an unambiguous machine-checkable acceptance rule, or if the difference between variants is smaller than the spread of the measurement, the study is published as "no difference shown" with no rating and no routing recommendation.
What one task costs on each side
Two kinds of worker do the same job. One is attached to an editor and covered by a flat subscription; the other is invoked as a headless worker and billed per token at $0.20 per million input tokens and $0.60 per million output tokens. Both write the same article or the same function.
The subscription side has a marginal API price of zero and a non-zero cost of quota and human attention: someone has to drive the session, and that time is the dominant term. The metered side has a measurable API price per task — around half a cent for a task that reads 15,000 tokens and writes 4,000 — and no driving time at all once the packet is written.
Neither column is comparable on its own. Put them on the same line and the comparison becomes honest: cost per accepted task, with human minutes converted to money at a published rate, and the quota consumption reported next to it.
⚠️ When this method does not work
- When acceptance is a matter of taste. Marketing copy, naming, visual design and "does this sound like us" have no machine check. Counting them as accepted or rejected produces precise numbers with no meaning.
- When the task set is not matched. If variant A got the four easy tasks and variant B got the four hard ones, the cost comparison measures the assignment, not the workers.
- When one side cannot be automated. An editor-bound agent cannot be dispatched headless, so its measurement always includes a human in the loop. That asymmetry is a finding to report, not a defect to engineer away by pretending the human is free.
- When the sample is small. Under 30 matched tasks, report the raw rows and stop. A two-task win is an anecdote with a spreadsheet around it.
- When the bill is not the constraint. If the real bottleneck is review capacity or a compliance gate, cheaper generation does not help, and optimizing cost per task optimizes the wrong thing.
What remains unmeasured
We have not yet run the 30-task comparison between our editor-bound agent and our metered worker. The protocol, the acceptance templates, the accounting tool and the cost estimate exist; the cohort does not. Until the rows exist, the honest status is "unrated" — not a ranking, not a percentage, not a savings claim.
The point of the incident is narrower than it looks. A number moved, we could see the number moving, and we still could not answer the only question that matters: what did we get for it. The method above exists so that the next time a counter runs out, the answer is already written down.
Sources
Source links and observation dates are recorded below. Review status is stated for each material.
🔗 Article connections
Open the full knowledge graph ↗Published article links. Select a node to open its article at the heading.
Show the connection graph
Article links · keyboard / text view
Same topic
- Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
- Read agent communication receipts without mistaking delivery for a human read
- Audit an AI technology radar against primary sources before promoting a story
- Run an AI coding harness evaluation offline before any live pilot
- DSHREM on a phone: SSH, tmux, and safe task handoff
- DSHREM owner practice: cwd, worktrees, WIP, and root boundaries
Next steps
Related reading
Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
Tvijo is developing a controlled workflow from a repository task to an accepted result, with permissions, resource limits and attributable evidence. PRIZMA describes the intended resource profile; RUNLEDGER connects the work to its checks a…
AI and process foundationRead agent communication receipts without mistaking delivery for a human read
Find the current stage of a swarm message without confusing durable storage, delivery, runtime receipt and viewing. This guide connects local checks, the private observation screen and distinct receipts, so a delivery acknowledgement does not become an unsupported claim that an action completed.
AI and process foundationAudit an AI technology radar against primary sources before promoting a story
Check a technology news item against its primary source, version and event date before deciding to adopt it. This guide follows a fixed ten-item sample, separates facts from interpretation and forecasts, and preserves limitations and the condition for checking the claim again.
💬 Comments