Audit an AI technology radar against primary sources before promoting a story
Check a technology news item against its primary source, version and event date before deciding to adopt it. This guide follows a fixed ten-item sample, separates facts from interpretation and forecasts, and preserves limitations and the condition for checking the claim again.
🎯 For Technology researchers and product teams deciding whether an AI tool or model deserves a tracked evaluation.
- Separate an aggregator’s publication date from the underlying event or release date.
- Keep sample counts fixed and classify primary-source support without replacing weak items.
- Mark publisher interpretation and analyst inference separately from facts.
A discovery newsletter is a lead list, not an authority. For the fixed ten-item New Runtime sample selected on 9 Nov 2026, the source audit classified 4 claims as confirmed, 3 partially confirmed, 2 stale, and 1 unverified; duplicate flags were counted separately and totaled 0. The fixed claim IDs were: NR-20261109-01, NR-20261109-02, NR-20261109-04, and NR-20261109-05 confirmed; NR-20261109-06, NR-20261109-07, and NR-20261109-09 partially confirmed; NR-20261109-03 and NR-20261109-10 stale; NR-20261109-08 unverified. These buckets sum to the ten selected records. A promotion rule requiring at least 8 confirmed, non-stale core claims was not met.
Goal
Check whether a claim is supported by a primary source and still current before adding it to a product or technology radar.
Review steps
- Freeze the item list before research. Keep every selected item in the denominator; do not replace weak matches with easier stories.
- Record the aggregator timestamp and the underlying event date separately. Check release versions or commit timestamps when available.
- Open the primary source: an official release, repository/tag, package registry, standards text, or original author’s report. An original author’s report supports what the author says, not independent performance claims.
- Record exact claim, source URL, version/commit, observed date, limitations, contradictions, and a concrete recheck condition.
- Label the publisher’s analysis and your own inference separately. Absence of official evidence is
unverified, not confirmation.
The audit found GPT-6.1 Astra shelved unverified: official OpenAI references reviewed named GPT-6 Astra and GPT-6.1 Sol, not the claimed combined model name. A secondary article alone did not resolve that mismatch. The sample also included release stories already superseded by newer official versions. See what MCP is for a separate protocol introduction; a product radar should not treat protocol familiarity as proof of product fit.
When this does not work
Produce one row per frozen claim with a status and a source trail; keep duplicate flags outside the disjoint status counts. Recheck when the official release page, repository tag, or relevant product capability changes. Common error: counting a publish date as the event date, treating an aggregator README as proof, or silently replacing an unverified record.
Claim scope: NR-USE-1–NR-USE-3, fixed sample selected 2026-11-09. These counts describe only those ten records and do not rate the source’s future issues or prove any product is safe or useful for Tvijo.
Read the frozen audit
curl --fail --silent --show-error https://tvijo.com/evals/tvijo-eval-comms-radar-20261109/new-runtime-audit.json -o tvijo-radar-audit.json
python3 -c 'import json; d=json.load(open("tvijo-radar-audit.json")); print(len(d["records"])); print(d["summary"]["classification_counts_exactly_over_selected_10"])'
Expected: ten records and counts confirmed=4, partially_confirmed=3, stale=2, unverified=1, duplicate=0. Nine items have supporting primary evidence, but some claims are partial or stale. This is not a longitudinal estimate of source accuracy. No automatic collector or paid job was enabled for it.
Phone and VS Code
From a phone, use Termius/SSH and Bash in a separate checkout; enter commands in the shell, not an agent task box. In VS Code Remote SSH use that checkout’s integrated terminal. Related setup: MCP in VS Code. Browser layout was checked at 320/390 pixels; a physical iPhone was not tested in this package.
Evidence
Checked 2026-11-09. Source files are available to maintainers in the private checkout containing this release; offline commands require that checkout, Node 26.7.0 and Python 3.12. Structured data and claim IDs · Primary-source audit · QM, reviewed commit · Promptfoo custom providers.
2026-11-09 UTC. The frozen pilot records and local scenarios listed above were checked. Until a separate publication receipt exists, a local build and screenshot do not establish that a public URL serves this release.
Practice
Run the audit-reading command above and preserve its output next to the check date. This reads published JSON without making a model request.
Check yourself
Choose an audit row and open its official primary source. Compare the event date with the news publication date, then check the reviewed commit. What does a product author’s description establish, and what requires a local experiment? Nine items with supporting sources do not mean nine fully confirmed current claims. Preserve contrary information and a reason to revisit the conclusion, even when the news item looks attractive.
Next
Continue with the linked guides: evaluation, delivery and source verification. An expanded sample is planned only. New reusable skills remain hypotheses: evaluate their benefit on different tasks with a frozen method and independent acceptance.
Sources
Pilot data, ten-item audit, Promptfoo documentation, QM reviewed commit. Sources establish provenance; applicability and benefit depend on the conditions of the specific check.
🔗 Article connections
Open the full knowledge graph ↗Published article links. Select a node to open its article at the heading.
Show the connection graph
Article links · keyboard / text view
Same topic
- Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
- Read agent communication receipts without mistaking delivery for a human read
- Run an AI coding harness evaluation offline before any live pilot
- DSHREM on a phone: SSH, tmux, and safe task handoff
- DSHREM owner practice: cwd, worktrees, WIP, and root boundaries
- DSHREM economy swarm: checkpoints, routes, and real cost evidence
Next steps
Related reading
Tvijo agent system: PRIZMA, RUNLEDGER and acceptance
Tvijo is developing a controlled workflow from a repository task to an accepted result, with permissions, resource limits and attributable evidence. PRIZMA describes the intended resource profile; RUNLEDGER connects the work to its checks a…
AI and process foundationRead agent communication receipts without mistaking delivery for a human read
Find the current stage of a swarm message without confusing durable storage, delivery, runtime receipt and viewing. This guide connects local checks, the private observation screen and distinct receipts, so a delivery acknowledgement does not become an unsupported claim that an action completed.
AI and process foundationRun an AI coding harness evaluation offline before any live pilot
Check the coding-agent evaluation path without a model request, then read eight real engineering pilot slots. This guide connects commands, independent artifact acceptance, full elapsed time and expense receipts, explaining why a saved response differs from a newly accepted result.
💬 Comments