How to evaluate agent quality, cost and control

This protocol is a proposal, not a completed benchmark. No score in this article ranks Tvijo, Codex, Claude, Gemini or Sakana. It describes evidence needed before publishing such a comparison.

Technology map · Exercises

Compare the same task

Freeze a task corpus, repository commit, environment image, allowed tools, acceptance commands, time observation method and budgets before running candidates. Separate model, provider, orchestration and permission preset. Compare a model alone and a managed workflow only when both conditions are explicitly described. Use fresh isolated workspaces and repeat each condition; publish the sample size, failures and uncertainty.

Start with synthetic documentation, coding and retrieval tasks. FullAccess and Max Speed are configuration names, not evidence of quality or safety. Experiments on risky presets belong in disposable sandboxes with fake secrets, no live accounts and bounded outbound access. Do not disable protection on a production host to manufacture a comparison.

Record outcomes and denominators

Provider price lists and requested model names do not replace usage receipts or served-model evidence. Report API costs, subscription allocation, compute and human review separately; unknown is not zero. Include pending and held spend. A timeout with unknown billing freezes further paid retries until reconciled.

What the 28 October pilot actually showed

Two different cheap-route advisory tasks cost $0.00139078856 and $0.00161825. One response claimed zero spend despite the ledger; the other inverted cost per accepted task. A Gemini 4 Argon review cost $0.021426 and reached its output limit. It also asserted that the home canonical choice prevents Workspace indexing; the supplied evidence does not establish that causal claim. These observations justify deterministic checks, not a model league table.

Total API spend was $0.02443503856, with held and pending balances reconciled to zero. This tiny heterogeneous pilot cannot estimate general quality, error rates or a cost advantage. A judge can be wrong too; preserve its answer and explain rejected claims.

METR is a separate measurement

METR time horizons relate agent success probability to task duration measured for human experts. They are not simply an agent's wall-clock runtime or a count of rule violations. A Tvijo study must use its own name unless it reproduces the relevant protocol and supports every methodological claim. This publication does not claim METR validation, certification or an equivalent time horizon.

Search and AI visibility study

Keep three ledgers: crawl/index eligibility, observed search traffic, and observed AI answers. For the citation panel, freeze a set of branded and unbranded questions in each language; record client, model, date, geography if known, search/tool availability and verbatim evidence URLs. Distinguish prompted mentions, unprompted mentions and links actually cited. Repeat on a defined schedule and report a rate only for that panel. One positive answer is not global recognition or training inclusion.

For Google indexing, define the eligible URL denominator from sitemap and robots, deduplicate canonicals, then inspect a documented sample or all URLs within the API quota. Report inaccessible and unknown states explicitly. The four URLs inspected on 28 October do not measure site-wide coverage.

Publication gates

1. Admit the route with enforceable spending bounds and provider identity.
2. Freeze the task and environment; isolate credentials and production services.
3. Run the declared repetitions; retain all failures and costs.
4. Have acceptance checked independently of the authoring worker.
5. Publish aggregate results, source hashes, limitations and conflicts of interest. Keep private traces authorized.
6. Re-run only when a declared version or protocol change warrants it; never select only favourable runs.

Sakana's own model information is a vendor source. Comparing those claims with our uncontrolled pilot would be invalid. The next phase is a bounded shared task set, not a star rating manufactured from catalogue prices.