Tvijo wiki
Catalog

Topics

Run an AI coding harness evaluation offline before any live pilot

Check the coding-agent evaluation path without a model request, then read eight real engineering pilot slots. This guide connects commands, independent artifact acceptance, full elapsed time and expense receipts, explaining why a saved response differs from a newly accepted result.

🎯 For Developers and evaluators who need to inspect a harness evaluation workflow without starting a provider request.

📚 You will learn
  • Install the pinned local evaluation CLI.
  • Run the recorded-fixture Promptfoo configuration with telemetry disabled and no model calls.
  • Distinguish a local fixture result from a live harness attempt.

As of 9 Nov 2026, Tvijo’s harness pilot includes a Promptfoo 0.125.1 offline configuration that reads sanitized recorded fixtures. The offline run checks parsing and acceptance reporting; it does not call a model and is not a new benchmark attempt. This distinction matters: a clean fixture run does not show that an agent completed a task.

Goal

Check that the offline evaluation path can read its recorded evidence and report expected fixture statuses, without making provider requests.

Practice

From the repository root, install the exact dependency versions in the lockfile, then run the offline configuration:

npm ci --prefix evals/tvijo-harness-pilot
cd evals/tvijo-harness-pilot
PROMPTFOO_DISABLE_TELEMETRY=1 PROMPTFOO_DISABLE_UPDATE=1 node node_modules/promptfoo/dist/src/main.js eval -c promptfooconfig.offline.yaml --no-cache

This configuration uses a local file provider, cache: false, one concurrency slot, one repetition, and four recorded fixtures. The expected output is a result for each fixture, with the recorded provenance marker; those rows are fixtures, not participant attempts. For a related introduction, see what MCP is and AI coding-agent cost per task.

Keep offline and live results separate

The live wrapper uses the existing native meter and a frozen run selection. It requires an explicit task-scoped environment guard, validates frozen source and independent-gate receipts, and stops when accounting is unresolved. A live run can contact a provider. This guide does not authorize or demonstrate one; do not remove those guards or substitute a fixture result for a live receipt.

When this does not work

A successful offline command should report four recorded fixture cases and no new provider request. If Node cannot find the pinned CLI, install from the lockfile before retrying. If package installation or the fixture command fails, record the setup error and stop; do not switch to the live wrapper to make the check look green.

Claim scope: EVAL-01, observed 2026-11-09. Applies to the frozen local offline configuration and recorded fixtures only. It makes no claim about live harness quality, real task completion, or production behavior.

Observed engineering pilot

Eight slots on two task fixtures: 4 PASS, 3 FAIL, 1 pre-model INFRA_ERROR. Hermes A remains in ITT as a controller infrastructure failure, not model quality. Acceptance below comes from independent artifact checks. Provider invoice is UNKNOWN; USD is usage times tariff. Saved offline responses are not counted among these eight slots.

Shell PASS/slots Median seconds API $ estimate API $/PASS Minutes/PASS
DSHREM 1/2 167.80 0.009505932 0.009505932 5.593
Pi 1/2 45.03 0.009056040 0.009056040 1.501
MiniSWE 2/2 51.59 0.008224230 0.004112115 0.860
Hermes 0/2 27.85 0.004547244 — —

Task rows, Q, versions and limits: Compare. Two task fixtures do not establish a universal winner or proven equivalence.

Check the published numbers

curl --fail --silent --show-error https://tvijo.com/evals/tvijo-eval-comms-radar-20261109/data.json -o tvijo-pilot-data.json
python3 -c 'import json; d=json.load(open("tvijo-pilot-data.json")); print(len(d["runs"]), sum(r["status"]=="PASS" for r in d["runs"]), d["API_total_usage_estimate_usd"])'

Expected: 8 4 0.031333446. Holds are settled and this live task token is closed. A new live study needs a new task ID, budget, frozen specification and independent admission. Reusing this task ID does not authorize another paid attempt.

Phone and VS Code

From a phone, use Termius/SSH and Bash in a separate checkout; enter commands in the shell, not an agent task box. In VS Code Remote SSH use that checkout’s integrated terminal. Related setup: MCP in VS Code. Browser layout was checked at 320/390 pixels; a physical iPhone was not tested in this package.

Evidence

Checked 2026-11-09. Source files are available to maintainers in the private checkout containing this release; offline commands require that checkout, Node 26.7.0 and Python 3.12. Structured data and claim IDs · Primary-source audit · QM, reviewed commit · Promptfoo custom providers.

2026-11-09 UTC. The frozen pilot records and local scenarios listed above were checked. Until a separate publication receipt exists, a local build and screenshot do not establish that a public URL serves this release.

Check yourself

Why should the offline check accept a recorded FAIL when it matches the expected fixture? Where does independent artifact acceptance come from? Find the slot with no model call and explain its place in ITT. Next inspect the cost ratio when no result was accepted: it must not become zero. Check the exact task ID and API-request count before reusing a number.

Next

Continue with the linked guides: evaluation, delivery and source verification. An expanded sample is planned only. New reusable skills remain hypotheses: evaluate their benefit on different tasks with a frozen method and independent acceptance.

Sources

Pilot data, ten-item audit, Promptfoo documentation, QM reviewed commit. Sources establish provenance; applicability and benefit depend on the conditions of the specific check.

🔗 Article connections

Open the full knowledge graph ↗

Published article links. Select a node to open its article at the heading.

Show the connection graph
100%
→ Linked articles← Linked from⋯ Same topic
Article links · keyboard / text view

💬 Comments

Comments are public and moderated. Do not post personal data.
  • No comments yet. Be the first!

Next steps