One new card from Hamel Husain. The Aug 13-20 window was well-covered by the five preceding daily runs, which together captured twelve cards across Tomasz Tunguz, Simon Willison, Aleyda Solis, Rand Fishkin, Chris Zukowski, Dan Shipper, Thorsten Ball, Kevin Indig, and others. Today's scan added one card from a significant update Husain published to an existing post.
Hamel Husain adds failure-mode discovery to the eval workflow
Husain updated "Evals Skills for Coding Agents" in August 2026 with two new skills that address a sequencing problem common in AI engineering teams: applying automated evaluation before the team has any taxonomy of what is going wrong.
The error-discovery skill reverses the standard order. A practitioner provides raw LLM outputs or traces. An agent builds a review application with intelligent sampling and, as the practitioner annotates examples, groups the annotations into named failure modes. The session ends with a typed taxonomy of failures rather than a score with no interpretable categories. A companion start routing skill handles the first decision: it reads the practitioner's situation and directs them to either an eval-audit path (if an existing eval pipeline is in place) or error-discovery (if raw unanalyzed traces are the starting point). The mechanism prevents a recurring anti-pattern where teams reach for automation before they understand what they are measuring. This extends Husain's earlier claim in Most LLM evaluation failures trace to product design problems upstream of the model, not model capability that evaluation failures usually trace to design problems upstream of the model, and connects to When a team says an AI product is hard to eval, the bottleneck is product design, not eval methodology where evals function as a signal about product quality. Teams must annotate raw LLM traces to discover failure modes before applying automated evaluation.
- Hamel Husain, Teams must annotate raw LLM traces to discover failure modes before applying automated evaluation. Teams must annotate raw LLM traces to name failure modes before applying automated evaluation; the error-discovery and start routing skills encode this sequence as an agent workflow.