Choose by workflow · checked 2026-11-07

Compare the whole configuration

A useful result names the model, harness, provider, task and acceptance check. We keep external studies, own evidence and evaluation hypotheses distinct.

This page explains how we read agent studies published by others and where their sources stop. The protocol for our own trials — what we compare, when a result counts as accepted and how cost is counted — is How we evaluate AI agents.

Local fixture evidence · 2026-11-10

What our local harness trials establish

Tvijo is evaluating its own harness. This engineering pilot covers local T1/T2/T3 fixtures, not SWE-bench or Terminal-Bench. This versioned reconciliation retains unknown task costs and acceptance. A completed worker turn does not establish independently accepted work.

Provider exports reconcile to USD 0.895596960 across two nonoverlapping account/key windows. This is neither a task cost nor an invoice or campaign settlement. Task-attributed cost, independent acceptance and allocated subscription seat cost remain unknown (null).

Configuration identity is partial. Returned names and aliases do not attest the model, provider or harness revision. Exact settings and exact patch attribution are unavailable for these observations. No quality index, ranking, stable multiplier or savings guarantee follows.

A comparison is a configuration

Record the task and acceptance check, model identifier, harness version, provider, effort, prompts, tools, environment, context limits, budget, date and repeats. Separate independent attempts from repair turns and judge votes. Unknown fields stay null or unreported.

Report accepted results alongside wall time, interventions, input/output/cache tokens, failed attempts and cost per accepted result. If there are no accepted results, the ratio is undefined. Subscription execution is not billed API spend. It still consumes a paid subscription and quota. Local hosting and free tiers have operating limits. Missing cost remains null.

Six studies, six different questions

These records document provenance and limits. They are not a pooled leaderboard. Current product releases in the overview are separate from the historical versions below.

External study · checked 2026-11-07

Joel Niklaus Coding Agent Bench

Publication / update
Unreported
Evaluation dates
Unreported
Tasks
Held-out SWE-bench Pro; ten harnesses × two models × 250 tasks
Independent attempts per pairing
1
Historical agent versions
Unreported / row-specific; never substituted with today’s release
Models
GLM-5.2; Gemma 4 26B-A4B
Provider
README describes Hugging Face Inference Providers for the baseline and local vLLM pools for Gemma/GLM; exact serving route is run-specific and unreported here.
Settings
One rollout per pairing in benchmark images. Current README pins thinking on server-side, with a separate thinking-off ablation. Historical per-run version, context/output and timeout stamps remain unreported here; README documents changed limits, so one numeric setting cannot represent every run.
Metrics
pass@1, confidence intervals, coverage, tokens and estimated cost
Limits
Current README is a moving snapshot, not a dated frozen study. Serving/proxy changes, context/output limits and infrastructure exclusions affect comparisons. Partial thinking-off ablations have uneven coverage. Exact historical versions remain null until run stamps are reviewed.

External study · checked 2026-11-07

Qihoo360 Harness-Bench V2

Publication / update
2026-10-15
Evaluation dates
Unreported
Tasks
V2 HTML/repository: 274 general + 26 multimodal tasks; the arXiv abstract still reports 106 tasks. Snapshot mismatch must be resolved before numerical reuse.
Independent attempts per pairing
1
Historical agent versions
Claude Code: 2.1.220; Codex: 0.145.0; OpenCode: 1.18.8; OpenClaw: 2026.5.26; Hermes: 2026.7.20; Goose: 1.44.0; Nanobot: 0.3.0
Models
GPT-5.6 Sol/Terra/Luna; GLM-5.3 / GLM-5.3-Flash; DeepSeek-V4 Pro/Flash; Kimi-K3; Gemma-4-31B; Qwen3.8-27B
Provider
Official provider APIs plus local Gemma/Qwen; exact endpoint identities unreported here
Settings
Temperature 1.0; 200K input threshold; 150 turns; model maximum output; native context management. General and multimodal coverage differs.
Metrics
Mean oracle score, full/threshold completion, trajectory length, API tokens/requests and estimated provider cost
Limits
V1 raw receipt is superseded by current V2. Abstract and HTML task counts disagree. One rollout gives no repeat variance. Locally hosted model operating costs are excluded from API cost, not zero.

External study · checked 2026-11-07

AIMultiple A-CODE-CLI

Publication / update
2026-06-29
Evaluation dates
2026-06-05 to 2026-06-08
Tasks
Ten web-development scenarios; backend and frontend smoke checks. Additional web research and memory probes use different models/protocols.
Independent attempts per pairing
Unreported; see settings
Historical agent versions
Claude Code: 2.1.165; Cline: 3.0.27; Codex: 0.140.0; Aider: 0.86.2; Gemini CLI: 0.26.0; Forge: 2.13.11; Goose: 1.37.0; Grok: 0.2.54; Junie: 26.06.01 (1831.35); Kiro CLI: 2.6.1; OpenCode: 1.17.7
Models
Sonnet 4.6 non-reasoning; Junie also uses GPT-4.1-mini
Provider
Native provider/OpenRouter; Claude Code subscription OAuth; Kiro hosted credits; Codex/Gemini via proxies
Settings
One-shot builds with default CLI settings. Selected backend evaluations rerun three times; that is not three independent builds for every pairing.
Metrics
Backend step-pass rate; UI smoke; combined 0.7 backend + 0.3 frontend; elapsed time, tokens, mixed billing costs
Limits
Codex and Gemini proxy adapters change behavior; Junie has a helper model. Research probes use native search models and N=1. Subscription/credits/API accounting cannot be treated as one token price. The public ZIP was retrieved after an initial HTTP403; its README states CC BY-NC 4.0. Raw dataset tables are not redistributed here. The example task is not the full task suite.

External study · checked 2026-11-07

Eishan Lawrence harness efficiency

Publication / update
2026-06-12
Evaluation dates
Unreported
Tasks
12 small Python tasks × 12 harness configurations (including Aider architect mode) × two model runs
Independent attempts per pairing
1
Historical agent versions
Unreported / row-specific; never substituted with today’s release
Models
deepseek/deepseek-v4-flash; nvidia/nemotron-3-ultra-550b-a55b:free
Provider
OpenRouter; upstream providers not pinned. Most use chat/completions; Claude Code uses the messages dialect.
Settings
One run per task/model/configuration in isolated Docker workspaces; deterministic oracles. Secondary model replication is not a repeat. Exact historical agent pins, effort and numeric budgets unreported. The paper also studies ten SWE-bench Lite tasks and injected context.
Metrics
Tokens per solved task, startup overhead and completion
Limits
Advertised repository returned HTTP 404 in the supplied receipt, limiting reproducibility. Small tasks mostly succeed; token ordering does not establish quality on larger repositories. A free tier is not zero resource or operating cost. Cache behavior depends on the harness/gateway/provider path and API dialect; auxiliary model calls remain in totals.

External study · checked 2026-11-07

Artificial Analysis Coding Agent Index v2.0

Publication / update
2026-10
Evaluation dates
Unreported
Tasks
FrontierCode v1.1 (100), Terminal-Bench 4.0 (66), SWE-Atlas-QnA-AA (124)
Independent attempts per pairing
3
Historical agent versions
Unreported / row-specific; never substituted with today’s release
Models
Unreported / row-specific
Provider
Unreported / row-specific
Settings
Settings, effort, agent version, model and provider are row-specific; there is no single shared configuration. Five judge votes on Q&A criteria are distinct from three attempts.
Metrics
Equal-weight benchmark index; per-attempt quality; elapsed time, tokens, API cost and integrity failures
Limits
This is a methodology, not one frozen leaderboard row. Historical versions/model/provider are null here; inspect the exact row before using a score. FrontierCode tasks are not public. Costs use API token pricing, not consumer subscriptions; missing telemetry is excluded, never filled with zero.

External study · checked 2026-11-07

Aider polyglot leaderboard

Publication / update
2025-11-20
Evaluation dates
Unreported
Tasks
225 Exercism exercises across C++, Go, Java, JavaScript, Python and Rust
Independent attempts per pairing
Unreported; see settings
Historical agent versions
Unreported / row-specific; never substituted with today’s release
Models
Unreported / row-specific
Provider
Unreported / row-specific
Settings
Row-specific version/commit, model, edit format, reasoning effort and command. Pass 2 repairs the first attempt after feedback; it is not an independent repeated trial.
Metrics
Percent correct, correct edit format, elapsed time and historical token cost
Limits
Last-updated date is not the date of every run. Example row: 2025-08-23, Aider 0.86.2.dev / commit 32faf82, gpt-5 (high), diff, command aider --model openai/gpt-5. Provider endpoint unreported. Do not replace row versions with current product versions.

What this edition does not establish

External scores do not transfer to Tvijo. The operator-reported isolated canary is own evidence, with private receipts and no independently reproduced quality comparison. Proposed integrations are hypotheses. A production activation or native swarm claim requires a separate accepted execution trace.

Terra: scoped native/code and public-source checks; human editorial moderation pending · Review by 2026-12-07