Tvijo wiki
Catalog

Topics

Audit an AI technology radar against primary sources before promoting a story

Check a technology news item against its primary source, version and event date before deciding to adopt it. This guide follows a fixed ten-item sample, separates facts from interpretation and forecasts, and preserves limitations and the condition for checking the claim again.

🎯 For Technology researchers and product teams deciding whether an AI tool or model deserves a tracked evaluation.

📚 You will learn
  • Separate an aggregator’s publication date from the underlying event or release date.
  • Keep sample counts fixed and classify primary-source support without replacing weak items.
  • Mark publisher interpretation and analyst inference separately from facts.

A discovery newsletter is a lead list, not an authority. For the fixed ten-item New Runtime sample selected on 9 Nov 2026, the source audit classified 4 claims as confirmed, 3 partially confirmed, 2 stale, and 1 unverified; duplicate flags were counted separately and totaled 0. The fixed claim IDs were: NR-20261109-01, NR-20261109-02, NR-20261109-04, and NR-20261109-05 confirmed; NR-20261109-06, NR-20261109-07, and NR-20261109-09 partially confirmed; NR-20261109-03 and NR-20261109-10 stale; NR-20261109-08 unverified. These buckets sum to the ten selected records. A promotion rule requiring at least 8 confirmed, non-stale core claims was not met.

Goal

Check whether a claim is supported by a primary source and still current before adding it to a product or technology radar.

Review steps

  1. Freeze the item list before research. Keep every selected item in the denominator; do not replace weak matches with easier stories.
  2. Record the aggregator timestamp and the underlying event date separately. Check release versions or commit timestamps when available.
  3. Open the primary source: an official release, repository/tag, package registry, standards text, or original author’s report. An original author’s report supports what the author says, not independent performance claims.
  4. Record exact claim, source URL, version/commit, observed date, limitations, contradictions, and a concrete recheck condition.
  5. Label the publisher’s analysis and your own inference separately. Absence of official evidence is unverified, not confirmation.

The audit found GPT-6.1 Astra shelved unverified: official OpenAI references reviewed named GPT-6 Astra and GPT-6.1 Sol, not the claimed combined model name. A secondary article alone did not resolve that mismatch. The sample also included release stories already superseded by newer official versions. See what MCP is for a separate protocol introduction; a product radar should not treat protocol familiarity as proof of product fit.

When this does not work

Produce one row per frozen claim with a status and a source trail; keep duplicate flags outside the disjoint status counts. Recheck when the official release page, repository tag, or relevant product capability changes. Common error: counting a publish date as the event date, treating an aggregator README as proof, or silently replacing an unverified record.

Claim scope: NR-USE-1–NR-USE-3, fixed sample selected 2026-11-09. These counts describe only those ten records and do not rate the source’s future issues or prove any product is safe or useful for Tvijo.

Read the frozen audit

curl --fail --silent --show-error https://tvijo.com/evals/tvijo-eval-comms-radar-20261109/new-runtime-audit.json -o tvijo-radar-audit.json
python3 -c 'import json; d=json.load(open("tvijo-radar-audit.json")); print(len(d["records"])); print(d["summary"]["classification_counts_exactly_over_selected_10"])'

Expected: ten records and counts confirmed=4, partially_confirmed=3, stale=2, unverified=1, duplicate=0. Nine items have supporting primary evidence, but some claims are partial or stale. This is not a longitudinal estimate of source accuracy. No automatic collector or paid job was enabled for it.

Phone and VS Code

From a phone, use Termius/SSH and Bash in a separate checkout; enter commands in the shell, not an agent task box. In VS Code Remote SSH use that checkout’s integrated terminal. Related setup: MCP in VS Code. Browser layout was checked at 320/390 pixels; a physical iPhone was not tested in this package.

Evidence

Checked 2026-11-09. Source files are available to maintainers in the private checkout containing this release; offline commands require that checkout, Node 26.7.0 and Python 3.12. Structured data and claim IDs · Primary-source audit · QM, reviewed commit · Promptfoo custom providers.

2026-11-09 UTC. The frozen pilot records and local scenarios listed above were checked. Until a separate publication receipt exists, a local build and screenshot do not establish that a public URL serves this release.

Practice

Run the audit-reading command above and preserve its output next to the check date. This reads published JSON without making a model request.

Check yourself

Choose an audit row and open its official primary source. Compare the event date with the news publication date, then check the reviewed commit. What does a product author’s description establish, and what requires a local experiment? Nine items with supporting sources do not mean nine fully confirmed current claims. Preserve contrary information and a reason to revisit the conclusion, even when the news item looks attractive.

Next

Continue with the linked guides: evaluation, delivery and source verification. An expanded sample is planned only. New reusable skills remain hypotheses: evaluate their benefit on different tasks with a frozen method and independent acceptance.

Sources

Pilot data, ten-item audit, Promptfoo documentation, QM reviewed commit. Sources establish provenance; applicability and benefit depend on the conditions of the specific check.

🔗 Article connections

Open the full knowledge graph ↗

Published article links. Select a node to open its article at the heading.

Show the connection graph
100%
→ Linked articles← Linked from⋯ Same topic
Article links · keyboard / text view

💬 Comments

Comments are public and moderated. Do not post personal data.
  • No comments yet. Be the first!

Next steps