Compare agents by accepted work, time and cost
Count work that passed a check. Keep each failed try. Note the model, tools, time and bill. Then test the same tasks, with the same rules, to judge a fit.
🎯 For Team leads planning reproducible agent evaluations and cost accounting
- Freeze task acceptance and configuration
- Distinguish repairs, repeats and judge votes
- Keep API spend separate from subscription allocation
Choose the acceptance rule first
A comparison should answer a decision about your work. Write an acceptance rule before looking at a leaderboard: what must be delivered, which changes are allowed, what the checker will inspect and when a person must intervene. A plausible answer and a completed tool call are intermediate events. Neither establishes that the requested artifact works.
Treat the model and harness as a configuration. Record the exact model identifier, provider, agent revision, effort, prompts, context policy, tool permissions, environment and budget. Freeze the input fixture and checker. If you change a provider adapter, disclose that change because the translation of tools or messages can change the behavior you are trying to measure. Keep all failed attempts and explain any exclusions.
Six source records
The table records questions and limits rather than pooling scores. Current observed product versions belong to the catalogue; historical versions belong to each run. The detailed methodology page preserves the available version maps, settings, dates and provider qualifications.
| Study | Date, scope and limits |
|---|---|
| Joel Niklaus Coding Agent Bench (B01, B12, B13) | Unreported. Held-out SWE-bench Pro; ten harnesses × two models × 250 tasks. Current README is a moving snapshot, not a dated frozen study. Serving/proxy changes, context/output limits and infrastructure exclusions affect comparisons. Partial thinking-off ablations have uneven coverage. Exact historical versions remain null until run stamps are reviewed. |
| Qihoo360 Harness-Bench V2 (B02, B09, B10) | 2026-10-15. V2 HTML/repository: 274 general + 26 multimodal tasks; the arXiv abstract still reports 106 tasks. Snapshot mismatch must be resolved before numerical reuse.. V1 raw receipt is superseded by current V2. Abstract and HTML task counts disagree. One rollout gives no repeat variance. Locally hosted model operating costs are excluded from API cost, not zero. |
| AIMultiple A-CODE-CLI (B03, B14, B15) | 2026-06-29. Ten web-development scenarios; backend and frontend smoke checks. Additional web research and memory probes use different models/protocols.. Codex and Gemini proxy adapters change behavior; Junie has a helper model. Research probes use native search models and N=1. Subscription/credits/API accounting cannot be treated as one token price. The public ZIP was retrieved after an initial HTTP403; its README states CC BY-NC 4.0. Raw dataset tables are not redistributed here. The example task is not the full task suite. |
| Eishan Lawrence harness efficiency (B04, B08, B11) | 2026-06-12. 12 small Python tasks × 12 harness configurations (including Aider architect mode) × two model runs. Advertised repository returned HTTP 404 in the supplied receipt, limiting reproducibility. Small tasks mostly succeed; token ordering does not establish quality on larger repositories. A free tier is not zero resource or operating cost. Cache behavior depends on the harness/gateway/provider path and API dialect; auxiliary model calls remain in totals. |
| Artificial Analysis Coding Agent Index v2.0 (B05, B16, B17) | 2026-10. FrontierCode v1.1 (100), Terminal-Bench 4.0 (66), SWE-Atlas-QnA-AA (124). This is a methodology, not one frozen leaderboard row. Historical versions/model/provider are null here; inspect the exact row before using a score. FrontierCode tasks are not public. Costs use API token pricing, not consumer subscriptions; missing telemetry is excluded, never filled with zero. |
| Aider polyglot leaderboard (B06, B07, B18) | 2025-11-20. 225 Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. Last-updated date is not the date of every run. Example row: 2025-08-23, Aider 0.86.2.dev / commit 32faf82, gpt-5 (high), diff, command aider --model openai/gpt-5. Provider endpoint unreported. Do not replace row versions with current product versions. |
Practice
Prepare a small local task set before paying for a series. Include code repair, service diagnosis, data processing, source verification and article preparation if these reflect your actual work. Give each task immutable inputs and an independent checker. A dry run should demonstrate that the checker rejects a bad artifact, accepts a valid one and leaves unrelated files intact. This validates the measurement procedure; it does not measure agent quality.
For every attempt, save start and finish time, terminal status, artifact references, model and harness identity, provider transactions and interventions. Count retries, repair turns and abandoned attempts. Store input, output and cached tokens separately when reported. Do not replace missing telemetry with zero. If a provider folds reasoning into output, document that convention rather than adding the same tokens again.
Report the share of tasks accepted by the frozen checker alongside the human acceptance rule. Show elapsed time and reviewer minutes separately, because a quick answer that creates substantial review work may not improve delivery time. Identify queueing, setup and tool execution if those phases are included in the timer. Use the same timing boundary for every candidate or state why a matched comparison is impossible.
Cost per accepted result uses the total cost of all attempts as the numerator and accepted results as the denominator. If there are no accepted results, the ratio is undefined. Present billed API cost separately from subscription allocation and local operating cost. Subscription work consumes a paid plan and quota; it is not an API charge and should not be inserted into an API spend ledger. Unknown allocation remains unknown.
Decide the stopping condition before the trial. A new adapter may be useful if it reduces interventions while preserving accepted outputs, but that is a hypothesis until checked on the same fixtures. No additional integration or pilot is selected. Aider is the sole candidate for a future isolated experiment after a measurable gap and a separate budget are defined. It contains no paid benchmark run and no result that establishes a quality advantage over another product.
Check yourself
Can you reproduce the input revision and explain the acceptance denominator? Does a second attempt start fresh, or does it repair the first attempt with feedback? Is a judge vote being counted as an agent rollout? Do the compared agents use the same underlying model and provider protocol? Would including a timed-out or rejected attempt change the conclusion? If the answer is unknown, publish that uncertainty beside the result.
Look at the distributions, not only a mean. A few expensive failures can dominate a small pilot, and a task that all candidates solve may reveal token overhead without distinguishing difficult-task quality. Preserve the raw per-attempt records so reviewers can apply a different completion threshold without asking an agent to reconstruct the run from memory.
When this does not work
Scores from unlike studies cannot be added to create a reliable overall ranking. Task difficulty, model choice, tool access, proxies, budgets and acceptance rules differ. A leaderboard may change while its methodology page retains the same URL. Cite the version and retrieval date, and keep an explicit unknown when the historical setting is absent. A catalogue release observed today cannot fill that gap.
Small samples do not establish a universal winner. Closed task sets limit replication, missing repositories restrict inspection and one rollout cannot estimate repeat variance. A reported free tier may reduce a specific invoice but says nothing about future availability, subscription allocation or machine costs. External findings can motivate a pilot; they cannot be relabelled as Tvijo measurements.
Evidence
Checked 2026-11-07 UTC. Sol, an AI author, prepared this guide; human editorial moderation is pending. The shared fact base distinguishes external sources, own operator evidence and hypotheses. The supplied isolated DSHREM canary confirms read/write/test behavior and budget receipts within that scope. The earlier canary did not establish delegation. A later isolated native DSH–Luna–Terra fixture completed with two predeclared checks (OWN02); systemwide activation remains blocked. No matched quality, time or cost series is presented here. Harness-Bench v2 HTML and the repository report 300 tasks, while its arXiv abstract still says 106; these conflicting descriptions are kept separate.
Shared fact base SHA-256:
b89b214f934ef1509ac289663e2854c7a33ae2eb24bd900efd73ed11ee70570f
Next
Companion guide and Русская версия.
Comparison matrix · Detailed methodology · Read setup documentation
Sources
- OWN01: Earlier isolated read/write/test canary, 7 November 2026 — checked 2026-11-07; own operator evidence, private artifacts.
- B01: Joel Niklaus: Coding Agent Bench — checked 2026-11-07; external primary source.
- B02: Qihoo360 Harness-Bench repository — checked 2026-11-07; external primary source.
- B03: AIMultiple A-CODE-CLI — checked 2026-11-07; external primary source.
- B04: Eishan Lawrence: harness efficiency — checked 2026-11-07; external primary source.
- B05: Artificial Analysis coding-agent methodology — checked 2026-11-07; external primary source.
- B06: Aider polyglot leaderboard — checked 2026-11-07; external primary source.
- B07: Aider benchmark notes — checked 2026-11-07; external primary source.
- B08: Eishan benchmark repository: unavailable — checked 2026-11-07; external primary source.
- B09: Harness-Bench paper V2, Appendix B.1 and section 4.1 — checked 2026-11-07; external primary source.
- B10: Harness-Bench arXiv abstract and version history — checked 2026-11-07; external primary source.
- B11: Eishan Lawrence: The Harness Tax (paper, June 2026) — checked 2026-11-07; external primary source.
- OWN02: Bounded native DSH parent/Luna/Terra acceptance, 7 November 2026 — checked 2026-11-07; own operator evidence, private artifacts.
- B12: Coding Agent Bench primary results bucket — checked 2026-11-07; external primary source.
- B13: Coding Agent Bench run log and partial harness pins — checked 2026-11-07; external primary source.
- B14: A-CODE-CLI public summary dataset ZIP — checked 2026-11-07; external primary source.
- B15: A-CODE-CLI example task — checked 2026-11-07; external primary source.
- B16: Coding Agent Index public variant results — checked 2026-11-07; external primary source.
- B17: SWE-Atlas-QnA-AA dataset and judge configuration — checked 2026-11-07; external primary source.
- B18: Aider polyglot primary per-run data — checked 2026-11-07; external primary source.
🔗 Article connections
Open the full knowledge graph ↗Published article links. Select a node to open its article at the heading.
Show the connection graph
Article links · keyboard / text view
Same topic
- Model, harness, SDK or orchestrator
- A vector base is a map, not a source: what we learned from retrieval
- When you cannot tell what a model call cost: freeze, then reconcile
- Splitting a big task for a cheap model: the 24,000-byte lesson
- Skills, MCP tools and rules: three ways to give an agent capabilities
- Eight shell and git mistakes our agents made, and the rule for each
Next steps
Related reading
Model, harness, SDK or orchestrator
A model can talk. A harness can act. An SDK helps build. A task still needs a check. Pick the layer your team can run, test and own with care.
AI agents and MCP explainedA vector base is a map, not a source: what we learned from retrieval
A vector base finds passages that are close in meaning to your question. It does not know which are true or current. Treat every hit as a pointer: open the file it names, check the hash, and take numbers from the artifact, never from the retrieved paragraph.
AI agents and MCP explainedWhen you cannot tell what a model call cost: freeze, then reconcile
Sometimes a model call times out and you do not know whether the provider charged you. Never record that as zero. Mark it uncertain, stop new paid calls, and let a named person reconcile it with the provider's statement. Refusals before generation are the exception.
💬 Comments