Gate every public claim against the source that backs it

01 Adopt the trace-or-stop rule before you write the gate.

The gate enforces a rule. State the rule first, in one line a person can hold in their head: if you cannot trace a claim to a source, write "unverified" and stop. This is the strictest discipline in the whole pipeline. Every metric, every "we drove" number, every round figure has to trace to a customer quote, a snapshot, a release, or a repo commit. If it does not, the claim is "unverified" and you either find the source or drop it. Composite quotes do not exist. Invented metrics do not exist. Round numbers that feel true do not exist. If you can't trace a claim to a source, write "unverified" and stop

The reason is asymmetric cost. Credibility compounds and decays the same way. Every fabricated claim that gets caught burns the trust the verified ones earned. The cost of writing "unverified" is small. The cost of getting caught making one up is the whole portfolio. If you can't trace a claim to a source, write "unverified" and stop

02 Make verification the easy path, not the afterthought.

Build the verification mechanism before you scale generation, not after. Systems that cannot quickly check their own output accumulate invisible drift. AI generates fluently and fails silently, so without a cheap, fast check path, there is no signal that separates a good output from a plausible-sounding bad one. Verification is a first-class design constraint in AI production systems, not an afterthought QA step

Easy means three things. The check runs automatically after every generation. It produces a binary or scalar signal. It costs less to run than regenerating the output would. When verification is easy, every failed output becomes a learning signal. When it is hard, every failed output is invisible. Verification is a first-class design constraint in AI production systems, not an afterthought QA step

This is the same conclusion operators reached from four separate lanes inside two weeks: research, agent frameworks, evals, and PM craft all landed on verification, not production, as the irreplaceable human job. Verification, not execution, is the irreplaceable human job

"You can outsource your thinking, but you can't outsource your understanding."

· Andrej Karpathy, Sequoia AI Ascent 2026 fireside, 2026-04-30 · “You can outsource thinking,…”

03 Compose the two gates so cited means cited-and-supported.

Run the gates in order. The order is load-bearing.

Cite-check runs first because there is no point checking whether a citation supports a claim until you know a citation exists. The two gates together assert one thing: the claim is cited AND the cite is plausibly true. A "PASS" at cite-check is not a "VERIFIED-TRUE." Cited is not the same as true, and treating it as the same is the most common way this whole discipline gets defeated.

04 Parse the asset into claim-and-path pairs, then open each path.

For each claim in the asset, find the nearest substrate path it cites. That gives you a set of (claim, cited-path) pairs. For each pair, open the substrate file at the cited path and check whether the claim's specific value appears there.

The proximity heuristic that maps a claim to its nearest cited path is the weak link. The nearest path is not always the semantic match, so the false-positive rate is non-trivial and the composite gate stays advisory until the mapping tightens. Treat a clean automated pass as a prompt to look harder at the pairs the heuristic was least sure about, not as a license to ship.

05 Define what counts as the value appearing in substrate.

Write the match rules down before you run them, so the gate is falsifiable and not a taste fight.

What does not count: a claim cited to a file that does not contain the value, even if the value is plausible. A claim whose value contradicts substrate (the asset says 70%, substrate says 65.1%, outside the 5% tolerance, flagged). A claim citing a path that does not exist, which cite-check catches earlier.

06 Build the judge as binary, one judge per failure mode.

The claim-verify check is an LLM-as-judge. Build it the way eval practitioners build judges that earn trust: binary pass/fail per failure mode, not a 1-to-5 score. A graded scale is a way of avoiding the decision. A binary judge forces a callable answer and a falsifiable accuracy number you can validate against human labels. Build LLM-as-judge as binary true/false, one judge per pesky failure mode, and validate against human labels

"1-2-3-4-5 is a weasel way of not making a decision."

· Hamel Husain, Lenny's Podcast, 2026-04-28 · “Build LLM-as-judge as binary…”

Keep the judge set small. Mature eval shops end with a handful of narrow judges, not dozens, because most failures get fixed at the source and never need a permanent check. Build LLM-as-judge as binary true/false, one judge per pesky failure mode, and validate against human labels Validate the judge against human-labelled traces with a confusion matrix, not a single agreement number. The off-diagonal cells are where a judge that says "pass" most of the time hides near-total failure on the long tail. And the trust math is the whole point of the gate. Evals are data analysis, single judge, binary rubrics, error analysis first

"When people lose trust in your evals, they lose trust in you."

· Hamel Husain, Lenny's Podcast, 2026-04-28 · “Build LLM-as-judge as binary…”

07 Do the error analysis before you trust any judge score.

Before you wire the judge into the pipeline, sit with a sample of real failures and code by hand what went wrong. This is the most-skipped step in AI evals and the one that returns the most per hour invested. Skip it and the judge measures the wrong things: it categorizes mismatches that do not happen at scale and misses the ones that do. Error analysis is the most-skipped step in AI evals and gives the most leverage per hour invested

For a claim-verify gate, the named failure modes are concrete. A round number with no source-system ID anywhere in substrate. A cite that was true 90 days ago but the substrate has since been refreshed and the asset still claims the old value. A quote attributed to a specific person that cannot be found in any source transcript. Code a sample, count the categories, then build the binary judge against the categories you actually found, not against generic quality axes. Evals are systematic data analysis on your LLM application, start with error analysis, not tests

"There's a common trap that a lot of people fall into because they jump straight to the test like, 'Let me write some tests,' and usually that's not what you want to do. You should start with some kind of data analysis to ground what you should even test."

· Hamel Husain, Lenny's Podcast, 2026-04-28 · “Evals are systematic data…”

08 Run the grader in its own context, separate from the generator.

The gate has to be independent of whatever produced the asset. Deploy the claim-verify check as a separate grader in its own context window, with access only to the output and the cited substrate, never to the generator's reasoning chain. A grader that shares context with the generator inherits the generator's blind spots and rubber-stamps the same invention twice. A separate grader agent in its own context window closes the output verification loop at production scale

"Agents do their best work when they know what 'good' looks like."

· Anthropic, Code with Claude, 2026-05-06 · “A separate grader agent…”

This is the clean role split that holds across PMM, growth, evals, and product: the substrate and the agents run the inner loop of production and retrieval, and humans run the outer loop of alignment and taste. The grader gate is where that boundary gets enforced on output. Substrate runs the loop; humans run alignment and taste

09 Keep a human verifying, because reliability quietly erodes the habit.

Verification is the irreplaceable human job, and the failure mode is that the human stops doing it. As the generator gets more reliable, review sessions shorten. Each skipped review feels harmless because the model is usually right. The aggregate effect is that you degrade the oversight mechanism at the exact point it still matters, and when failure does come, there is no review habit left to catch it. As coding agents become more reliable, review discipline erodes and the failure mode becomes invisible Verification, not execution, is the irreplaceable human job

"Those things have started to blur for me already, which is quite upsetting."

· Simon Willison, Vibe coding and agentic engineering, 2026-05-06 · “As coding agents become…”

Name one owner for the gate. Eval work is judgment work, and a committee debating what counts as a failure never ships. One trusted-taste arbiter ships, and the rubric tightens through use. Appoint one trusted-taste expert as the eval benevolent dictator, committees stall the loop

"When you're doing this open coding, a lot of teams get bogged down in having a committee... You can appoint one person whose taste that you trust."

· Hamel Husain, Lenny's Podcast, 2026-04-28 · “Appoint one trusted-taste expert…”

10 Fix fabrication at the input, not just at the output.

The gate catches fabrication after it happens. Cut the rate of fabrication going in by changing how the model runs. Running AI across three or more inputs at once, with thin or contradictory context, produces fabricated output, not analysis errors, because the model fills the gaps. Running AI on multiple inputs simultaneously without structured validation checkpoints produces fabricated output, not analysis errors

"AI needs A LOT of help to do a good job (way more than the average person realizes)."

· Anthony Pierri, AI needs A LOT of help to do a good job, 2026-05-10 · “Running AI on multiple…”

Two input-side fixes. First, add an explicit validate-against-sources checkpoint at each stage of a multi-input workflow, so the gap-filling loop gets interrupted before it compounds downstream. Running AI on multiple inputs simultaneously without structured validation checkpoints produces fabricated output, not analysis errors Second, do not batch heterogeneous inputs through one prompt. Batch-processing transcripts produces quotes that cannot be traced to any source file. Run a dedicated skill per input instead, at roughly three minutes per file, which tightens the context and makes spot-checking tractable: verify one file and you know the method held. Batch-processing interview transcripts with Claude produces fabrications; reliable output requires a dedicated per-transcript skill at three minutes per file

The deeper lever is the substrate itself. Most of the output quality comes from the knowledge fed in, not from the sophistication of the agents. A weak knowledge base produces weak, generic, untraceable content at high volume, so a clean canonical artifact per entity is itself a fabrication control. 90% of AI content system output quality comes from the knowledge fed in, not from agent sophistication. One canonical artifact, many consumers.

"90% of output quality comes from what you feed the system, not from the sophistication of the agents"

· Maja Voje, Content Engineering, 2026-05-01 · “90% of AI content…”

11 Surface uncertainty instead of hiding it.

Where a claim cannot be cleanly verified, do not let the asset state it with false confidence. Show the uncertainty. Hidden confidence trains the reader to either verify everything or trust nothing, and both destroy the asset's value. Transparent uncertainty trains the reader to verify the low-confidence claims and trust the high-confidence ones. Show model uncertainty in the UI; opaque confidence destroys trust

"Most teams hide model uncertainty from users. This is a miss. Showing uncertainty (confidence scores, multiple hypotheses, 'I'm not sure') builds trust and surfaces where the model is weak."

· Aishwarya Naresh Reganti, Lenny's Podcast, 2026-04-28 · “Show model uncertainty in…”

The gate's own output should do the same. A per-mismatch report names the line, the claim, the cited path, and what the substrate actually says, rather than collapsing to a single green check that hides the long tail.

12 Sweep for hallucinated claims about you, not just claims by you.

The gate so far defends outbound assets. There is an inbound version. Synthesis surfaces have a thin verification layer for sparsely-covered claims, so a single seeded fake claim can self-confirm by citing the source that introduced it. A competitor or a random actor can plant a false claim about your pricing, deliverability, or compliance, and the surface will repeat it back as fact. A single seeded fake claim can self-confirm in AI Overviews

Add a periodic brand-protection sweep to the cadence: query the AI surfaces for claims about your brand, check each surfaced claim against your own substrate, and treat a hallucinated claim as a comms escalation with its own response-time clock, separate from search-results monitoring. A single seeded fake claim can self-confirm in AI Overviews