SPIDER: a source-to-evidence workflow
Define the sources, preserve provenance and make missing data visible.
Collection has a boundary
Start with an allowlist of public sources and their permitted access methods. Record retrieval time, publication time, original identifier and extraction version. A crawler is not permission to bypass access restrictions.
Keep states explicit
Fresh, stale, unavailable and zero convey different information. Deduplicate using source identity and content version. Retain the original source link so a reader can inspect the record.
From a diagram to a service
The interactive graph in this wiki illustrates the workflow; it is not a live scan. A service integration additionally needs a source contract, schedule, error behavior and acceptance sample.