a builder's codex
codex · playbooks · Claim-verify gate playbook

Claim-verify gate playbook

Annotated process · two-gate system

The Claim-Verify System

Gate 6

Substrate-cite-check

"Is there a substrate citation near this claim?" Catches claims that float free of any source. A claim without a citation is a guess with formatting.

Gate 7

Claim-verify

"Does the cited substrate actually contain the claimed value?" Catches plausible-but-wrong citations. A citation is not a verified claim. The cite must contain the value.

03

Parse the asset

Extract (claim, nearest-substrate-path) pairs. Every external-facing claim with a specific number, date, named entity, or source-system ID gets a pair.

04

Open the substrate file

Check if the claim's specific value appears in that file. Exact match required for dates and named entities. Within 5% tolerance for percentages and quantity units.

Gate

VERIFIED or CITE-VALUE-MISMATCH

Verified: value appears in substrate within tolerance. CITE-VALUE-MISMATCH: value is absent or outside tolerance. Flag with specific path and discrepancy. Do not publish mismatches.

Gate 6 + Gate 7 together: "the claim is cited AND the cite is plausibly true" →

A two-gate substrate-fidelity defense for any system that grounds external assets in citable knowledge. Gate 6 (cite-check) catches claims with no cite. Gate 7 (claim-verify) catches claims cited to a path that does not actually contain the claimed value. Together they close the substrate-fidelity concern that pure citation existence misses.

Source synthesis: anti-fabrication ladder; evidence ladder (verified / self-reported / contextual / indirect / direct); Wikipedia AI Cleanup field guide; Hamel Husain + Shreya Shankar on per-claim factuality eval (Evals are systematic data analysis on your LLM application, start with error analysis, not tests, Build LLM-as-judge as binary true/false, one judge per pesky failure mode, and validate against human labels).

When to use

How the two gates compose

Together: "the claim is cited AND the cite is plausibly true."

Process

  1. Parse the asset for (claim, nearest-substrate-path) pairs.
  2. For each pair, open the substrate file at the cited path.
  3. Check if the claim's specific value appears in the substrate.
  4. If yes → verified. If no → flag as CITE-VALUE-MISMATCH.

What counts as "value appears in substrate"

What does NOT count

Output

Per-mismatch report: line, claim, cited path, what substrate actually says. PASS / FAIL summary. Exit code 0 if all cited claims verify; 1 if mismatches detected; 2 on file/config error.

Exit policy

Zero tolerance default. Substrate-fidelity failures cannot be averaged away.

Composes with

Worked example

A canonical positioning statement claims competitor X has "no voice agent" and cites a battle-card file. Claim-verify opens the cited card, searches for "no voice agent", not found. Searches "voice agent" within 30 chars of "no", finds "AI for post-call" instead. Flags CITE-VALUE-MISMATCH: substrate says competitor has post-call AI, not "no voice agent." The error would otherwise have shipped externally.

Common failure modes

Calibration signal

Track under taste-type "substrate-fidelity-truth." Brier signal: post-publish corrections needed where substrate did not support the claim.

Open the interactive view →