Recorded case

Measure cost per accepted result

Small task benchmarks illustrate why task class and acceptance matter more than a cheap model label.

Reviewed 15 October 2026 · scope: verified

Recorded method and findings

On 13 October, ten read-only repository tasks all met preregistered deterministic answers. Their recorded total was $0.01859652 across 53 calls, and each task receipt matched its ledger entries. These were search, counting and arithmetic tasks, not code changes.

Recorded method and findings

On 15 October, a separate extraction benchmark accepted one worker at completeness 0.923 and rejected another at 0.346 because its output was incomplete. The costs were $0.03505 and $0.01320 respectively. Paying less for a rejected result did not make it the better completion.

Practical checklist

Compare accepted results within the same task class and scope. Record failures, retries, elapsed time and human review effort. An unknown cost is unresolved accounting, not zero.

Sources & scope

Ten read-only repository tasks: accepted result benchmark 2026-10-13
Preregistered extraction completeness benchmark 2026-10-15

Keep exploring

AI orchestration: from a prompt to an accepted resultA practical design for delegation, evidence and recovery.Shared memory without shared mistakesKeep facts, hypotheses and permissions distinct across agent sessions.Engineering that can show its workTVIJO connects operators, bounded agent work and knowledge that can be checked and reused.AGENTORCH: the orchestrator has a concrete jobAGENTORCH names the orchestration role: plan, assign, observe, reconcile and accept.The task packet is the interfaceA good task describes a verifiable outcome rather than an invitation to improvise without limits.AGENTJUDGE: acceptance follows evidenceA reviewer checks the task, the artifact version and each acceptance criterion.Budgets and outages belong in the state machineA reliable operator view distinguishes working, waiting, failed, blocked and accounting uncertainty.Keep VECTORBASE, evaluate retrievalUse a private source-linked index to find evidence; do not treat search results as authority.SPIDER: evidence collection and useful progressThe first implementation reads an explicit project-evidence list and produces a dated operator digest without model calls.DSHREMBOT and tmux: one operator viewAuthenticated project commands, budget visibility and explicit schedule controls connect the operator to existing execution lanes.The AI worker–Hugging Face incident: learn from controlsThe incident reports support specific engineering lessons, not a controlled ranking of our system.What the supplied AI worker screenshot actually provesThe screenshot records a safeguard-triggered switch in one session; it does not explain the exact trigger. The observation is the supplied screenshot dated 15 October 2026.Training our own model: a gated research directionA proprietary model is an option to test after retrieval, prompts and routing have measurable limits.A bilingual knowledge product with visible limitsPublish useful explanations first, then expand languages when the evidence and workflow are ready.Free learning, custom projects and future challengesTurn evidence-backed engineering into clear offers and a useful developer community.Trust the sender, not a public checksumBind the complete message to an explicitly trusted authority.