One card from one operator pair. A thin but clean day: the 7-day window was covered heavily in yesterday's batch, and the one genuinely new, uncaptured finding is worth adding on its own.
Eval failure modes, the class agents cannot reach
Hamel Husain and Shreya Shankar published "Advanced evals: How to find (and fix) hidden AI failures in your product" in Lenny's Newsletter on September 22, 2026. The core contribution is a named failure mode with a mechanism.
Automated eval agents cannot detect failures that require your product's definition of good user experience captures it. Husain and Shankar call it criteria drift: the process of reviewing real traces actively changes your definition of what constitutes a failure. Agents trained or prompted before that review will optimize for the prior, ungrounded definition. The gap between what you thought success meant and what you learned it means is the drift. Agents cannot close that gap on their own. They pattern-match against explicit criteria; they cannot derive implicit product judgment from traces alone.
The mechanism is sequencing. Human annotation must precede automation. Not because agents are weak, but because the criteria agents need to apply don't fully exist before humans have reviewed enough examples to stabilize them. The leasing-assistant example makes the failure concrete. An assistant responding pleasantly to a prospect who couldn't afford the listed unit was being logged as successful by automated systems. Human reviewers flagged it immediately. The product goal was facilitating sales through objection handling. That criterion was never given to the agent. It had no way to derive it from the trace.
"Agents are often faster than humans at spotting obvious issues and find patterns we might miss. But they are far less reliable when a failure depends on your definition of a good product experience."
This card connects directly to the three earlier Husain cards in the library. Teams must annotate raw LLM traces to discover failure modes before applying automated evaluation established the sequencing rule: annotate before automating. Criteria-drift is the failure mode that explains why that sequencing rule exists. Eval tooling anchors teams on generic automated metrics before they have examined enough outputs to know what actually fails documented the generic anchor-metric problem. Criteria-drift is the mechanism underneath it. Most LLM evaluation failures trace to product design problems upstream of the model, not model capability addressed upstream product design failures. The leasing-assistant example sits at that boundary: a product-design failure and a criteria-drift failure simultaneously. The four cards now form a complete causal chain.
- Hamel Husain (with Shreya Shankar), Automated eval agents cannot detect failures that require your product's definition of good user experience. AI eval agents cannot identify failures that depend on your product's definition of good user experience; human annotation must precede automation to ground the criteria agents are asked to apply.