Tvijo wiki
Catalog

Topics

What an AI agent really costs: the accepted-task method

A 2,000-minute CI allowance hit 90% in three and a half days. The fix is not a cheaper model but a different denominator: cost per accepted task, checked by a script.

🎯 For Teams paying for AI agents who suspect the bill and the value have drifted apart.

📚 You will learn
  • Replace token or subscription counters with cost per accepted task
  • Write the acceptance rule and the death criterion before the first run
  • Report subscription quota and human minutes instead of calling a paid plan free
  • Recognise five situations where the method produces precise numbers with no meaning

A 2,000-minute monthly CI allowance was 90% gone after three and a half days, and nobody could say what the work that burned it had produced. That is the whole problem in one sentence: we could count the minutes, the tokens and the subscription, but we could not count accepted work. This page is the method we now use to count it, written after measuring our own case on 4 November 2026.

The short answer: measure the cost of one accepted task — money, human minutes, rework and overhead divided by the number of deliverables that passed a machine check on the first attempt. Counters such as tokens, requests, "credits", CI minutes or a flat monthly price are all inputs to that number, never the number itself.

Why the obvious counters mislead

A subscription makes every single run look free, because the payment already happened last month. A metered API makes every single run look expensive, because each call has a price attached. Both views are wrong in the same direction: they price the input and stay silent about the result.

Our own evidence, measured on 4 November 2026 from repository run metadata (not from a dashboard screenshot):

Observation Value
Monthly included CI minutes 2,000
Consumed when the warning arrived 1,812 (90%)
Days used of the 30-day cycle 3.5
Runs of one workflow in two days 109
Average duration of that workflow 14.1 minutes

The expensive part was not the model. It was 109 pushes in two days, each starting a 14-minute job that installed dependencies and ran a full test suite. Two of those runs per push, because the same commit triggered both a branch push and a pull-request event, and nothing cancelled the older one. The agent wave produced the pushes; the bill arrived in a currency nobody was watching.

The method

  1. Freeze the task before you dispatch it. Write down the input, the expected output, the acceptance rule, the attempt limit and the date. Everything that judges the outcome is written before the outcome exists.
  2. Accept with a script, not with an opinion. For a draft, the check is that every claim matches a pre-verified fact pack and every link returns 200. For extraction, it is an exact match against a frozen schema. For code, it is the named test file plus the linter. A model may comment on quality, but it does not decide acceptance.
  3. Record one row per attempt. Task, class, variant, accepted, API spend, human minutes, elapsed seconds, overhead, evidence reference, timestamp, and the billing kind (subscription or metered). A subscription row carries api_usd = 0.0 and the consumed quota units — writing "free" is a defect, not a shortcut.
  4. Count only accepted work. A rejected attempt is a cost with no deliverable: it stays in the spend column and never enters the denominator of finished tasks.
  5. Publish at least 30 matched tasks. Thirty is a minimum sample gate, not statistical significance. Ten per task class, so one class failing does not get reported as the general verdict.
  6. Announce the death criterion first. Ours: if fewer than 80% of tasks have an unambiguous machine-checkable acceptance rule, or if the difference between variants is smaller than the spread of the measurement, the study is published as "no difference shown" with no rating and no routing recommendation.

What one task costs on each side

Two kinds of worker do the same job. One is attached to an editor and covered by a flat subscription; the other is invoked as a headless worker and billed per token at $0.20 per million input tokens and $0.60 per million output tokens. Both write the same article or the same function.

The subscription side has a marginal API price of zero and a non-zero cost of quota and human attention: someone has to drive the session, and that time is the dominant term. The metered side has a measurable API price per task — around half a cent for a task that reads 15,000 tokens and writes 4,000 — and no driving time at all once the packet is written.

Neither column is comparable on its own. Put them on the same line and the comparison becomes honest: cost per accepted task, with human minutes converted to money at a published rate, and the quota consumption reported next to it.

⚠️ When this method does not work

  • When acceptance is a matter of taste. Marketing copy, naming, visual design and "does this sound like us" have no machine check. Counting them as accepted or rejected produces precise numbers with no meaning.
  • When the task set is not matched. If variant A got the four easy tasks and variant B got the four hard ones, the cost comparison measures the assignment, not the workers.
  • When one side cannot be automated. An editor-bound agent cannot be dispatched headless, so its measurement always includes a human in the loop. That asymmetry is a finding to report, not a defect to engineer away by pretending the human is free.
  • When the sample is small. Under 30 matched tasks, report the raw rows and stop. A two-task win is an anecdote with a spreadsheet around it.
  • When the bill is not the constraint. If the real bottleneck is review capacity or a compliance gate, cheaper generation does not help, and optimizing cost per task optimizes the wrong thing.

What remains unmeasured

We have not yet run the 30-task comparison between our editor-bound agent and our metered worker. The protocol, the acceptance templates, the accounting tool and the cost estimate exist; the cohort does not. Until the rows exist, the honest status is "unrated" — not a ranking, not a percentage, not a savings claim.

The point of the incident is narrower than it looks. A number moved, we could see the number moving, and we still could not answer the only question that matters: what did we get for it. The method above exists so that the next time a counter runs out, the answer is already written down.

Sources

Source links and observation dates are recorded below. Review status is stated for each material.

  1. GitHub Docs: About billing for GitHub Actions · Checked 2026-11-04 · Included minutes per plan, what counts as a billable minute, and what happens when included usage runs out.
  2. GitHub Docs: Control the concurrency of workflows and jobs · Checked 2026-11-04 · Concurrency groups and cancel-in-progress, used to stop superseded runs from consuming minutes.
  3. AI OS evidence: 1,812 of 2,000 CI minutes in 3.5 days · Checked 2026-11-04 · Internal measured receipt: per-day, per-workflow and per-branch minute table computed from run metadata; the 1,812-of-2,000 figure comes from the provider alert, the breakdown from the API.

🔗 Article connections

Open the full knowledge graph ↗

Published article links. Select a node to open its article at the heading.

Show the connection graph
100%
→ Linked articles← Linked from⋯ Same topic
Article links · keyboard / text view

💬 Comments

Comments are public and moderated. Do not post personal data.
  • No comments yet. Be the first!

Next steps