Run an AI coding harness evaluation offline before any live pilot
Check the coding-agent evaluation path without a model request, then read eight real engineering pilot slots. This guide connects commands, independent artifact acceptance, full elapsed time and expense receipts, explaining why a saved response differs from a newly accepted result.
🎯 For Developers and evaluators who need to inspect a harness evaluation workflow without starting a provider request.
- Install the pinned local evaluation CLI.
- Run the recorded-fixture Promptfoo configuration with telemetry disabled and no model calls.
- Distinguish a local fixture result from a live harness attempt.
As of 9 Nov 2026, Tvijo’s harness pilot includes a Promptfoo 0.125.1 offline configuration that reads sanitized recorded fixtures. The offline run checks parsing and acceptance reporting; it does not call a model and is not a new benchmark attempt. This distinction matters: a clean fixture run does not show that an agent completed a task.
Goal
Check that the offline evaluation path can read its recorded evidence and report expected fixture statuses, without making provider requests.
Practice
From the repository root, install the exact dependency versions in the lockfile, then run the offline configuration:
npm ci --prefix evals/tvijo-harness-pilot
cd evals/tvijo-harness-pilot
PROMPTFOO_DISABLE_TELEMETRY=1 PROMPTFOO_DISABLE_UPDATE=1 node node_modules/promptfoo/dist/src/main.js eval -c promptfooconfig.offline.yaml --no-cache
This configuration uses a local file provider, cache: false, one concurrency slot, one repetition, and four recorded fixtures. The expected output is a result for each fixture, with the recorded provenance marker; those rows are fixtures, not participant attempts. For a related introduction, see what MCP is and AI coding-agent cost per task.
Keep offline and live results separate
The live wrapper uses the existing native meter and a frozen run selection. It requires an explicit task-scoped environment guard, validates frozen source and independent-gate receipts, and stops when accounting is unresolved. A live run can contact a provider. This guide does not authorize or demonstrate one; do not remove those guards or substitute a fixture result for a live receipt.
When this does not work
A successful offline command should report four recorded fixture cases and no new provider request. If Node cannot find the pinned CLI, install from the lockfile before retrying. If package installation or the fixture command fails, record the setup error and stop; do not switch to the live wrapper to make the check look green.
Claim scope: EVAL-01, observed 2026-11-09. Applies to the frozen local offline configuration and recorded fixtures only. It makes no claim about live harness quality, real task completion, or production behavior.
Observed engineering pilot
Eight slots on two task fixtures: 4 PASS, 3 FAIL, 1 pre-model INFRA_ERROR. Hermes A remains in ITT as a controller infrastructure failure, not model quality. Acceptance below comes from independent artifact checks. Provider invoice is UNKNOWN; USD is usage times tariff. Saved offline responses are not counted among these eight slots.
| Shell | PASS/slots | Median seconds | API $ estimate | API $/PASS | Minutes/PASS |
|---|---|---|---|---|---|
| DSHREM | 1/2 | 167.80 | 0.009505932 | 0.009505932 | 5.593 |
| Pi | 1/2 | 45.03 | 0.009056040 | 0.009056040 | 1.501 |
| MiniSWE | 2/2 | 51.59 | 0.008224230 | 0.004112115 | 0.860 |
| Hermes | 0/2 | 27.85 | 0.004547244 | — | — |
Task rows, Q, versions and limits: Compare. Two task fixtures do not establish a universal winner or proven equivalence.
Check the published numbers
curl --fail --silent --show-error https://tvijo.com/evals/tvijo-eval-comms-radar-20261109/data.json -o tvijo-pilot-data.json
python3 -c 'import json; d=json.load(open("tvijo-pilot-data.json")); print(len(d["runs"]), sum(r["status"]=="PASS" for r in d["runs"]), d["API_total_usage_estimate_usd"])'
Expected: 8 4 0.031333446. Holds are settled and this live task token is closed. A new live study needs a new task ID, budget, frozen specification and independent admission. Reusing this task ID does not authorize another paid attempt.
Phone and VS Code
From a phone, use Termius/SSH and Bash in a separate checkout; enter commands in the shell, not an agent task box. In VS Code Remote SSH use that checkout’s integrated terminal. Related setup: MCP in VS Code. Browser layout was checked at 320/390 pixels; a physical iPhone was not tested in this package.
Evidence
Checked 2026-11-09. Source files are available to maintainers in the private checkout containing this release; offline commands require that checkout, Node 26.7.0 and Python 3.12. Structured data and claim IDs · Primary-source audit · QM, reviewed commit · Promptfoo custom providers.
2026-11-09 UTC. The frozen pilot records and local scenarios listed above were checked. Until a separate publication receipt exists, a local build and screenshot do not establish that a public URL serves this release.
Check yourself
Why should the offline check accept a recorded FAIL when it matches the expected fixture? Where does independent artifact acceptance come from? Find the slot with no model call and explain its place in ITT. Next inspect the cost ratio when no result was accepted: it must not become zero. Check the exact task ID and API-request count before reusing a number.
Next
Continue with the linked guides: evaluation, delivery and source verification. An expanded sample is planned only. New reusable skills remain hypotheses: evaluate their benefit on different tasks with a frozen method and independent acceptance.
Sources
Pilot data, ten-item audit, Promptfoo documentation, QM reviewed commit. Sources establish provenance; applicability and benefit depend on the conditions of the specific check.
🔗 Article connections
Open the full knowledge graph ↗Published article links. Select a node to open its article at the heading.
Show the connection graph
Article links · keyboard / text view
Same topic
- Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
- Read agent communication receipts without mistaking delivery for a human read
- Audit an AI technology radar against primary sources before promoting a story
- DSHREM on a phone: SSH, tmux, and safe task handoff
- DSHREM owner practice: cwd, worktrees, WIP, and root boundaries
- DSHREM economy swarm: checkpoints, routes, and real cost evidence
Next steps
Related reading
Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
Tvijo is developing a controlled workflow from a repository task to an accepted result, with permissions, resource limits and attributable evidence. PRIZMA describes the intended resource profile; RUNLEDGER connects the work to its checks a…
AI and process foundationRead agent communication receipts without mistaking delivery for a human read
Find the current stage of a swarm message without confusing durable storage, delivery, runtime receipt and viewing. This guide connects local checks, the private observation screen and distinct receipts, so a delivery acknowledgement does not become an unsupported claim that an action completed.
AI and process foundationAudit an AI technology radar against primary sources before promoting a story
Check a technology news item against its primary source, version and event date before deciding to adopt it. This guide follows a fixed ten-item sample, separates facts from interpretation and forecasts, and preserves limitations and the condition for checking the claim again.
💬 Comments