Four cards from three operators. Two themes from the engineering side of AI-native work, one from the organizational side. Each stands on its own, but Thorsten Ball's structural claim about bug classes sits underneath everything: if AI changes which errors dominate, that changes how you evaluate outputs and how you explain the change to executives.
Where software fails when AI handles coding
Thorsten Ball published "What I believe about the future of software development" on September 19. The core claim in As AI handles coding reliably, most software bugs shift from implementation errors to specification errors is structural: as AI coding reliability rises, the dominant bug class shifts from implementation errors to specification errors. The old failure was wrong code. The new failure is a correct implementation of the wrong ask.
Ball's mechanism is clean. Implementation bugs live in the gap between intent and machine execution. AI narrows that gap. Specification bugs survive AI entirely because they originate before code generation begins. No amount of AI coding capability eliminates the error of asking for the wrong thing. His framing from direct experience shipping with Amp:
"Most bugs won't be 'coding' bugs...they'll be 'you asked for the wrong thing' bugs"
This card connects to Ball's three existing library entries. Engineers who dismiss AI-generated code as low-quality apply the same professional bias that causes domain experts to over-intervene within their specialty documented how humans over-correct AI output they cannot evaluate at speed. Engineering ownership begins with naming the actual problem, not accepting the proposed solution established the ownership principle. Specification clarity is what those two cards were pointing toward without naming it.
- Thorsten Ball, As AI handles coding reliably, most software bugs shift from implementation errors to specification errors. As AI coding reliability rises, specification errors replace implementation errors as the dominant bug class; the failure is asking for the wrong thing, not writing wrong code.
Annotation quality in AI evals: two fixes from one source
Hamel Husain and Shreya Shankar published "AI Evals: Everything You Need to Know" on hamel.dev on September 18. The piece is a distillation from teaching 5,000 engineers and PMs. Two distinct claims are new to the library and both are immediately applicable.
Binary pass/fail annotation produces better quality than Likert scales by forcing judgment on uncertain cases addresses annotation scale design. Likert scales degrade quality by giving annotators a middle-value escape hatch: a "3 of 5" encodes uncertainty as data. Binary pass/fail removes the escape hatch. Husain and Shankar are direct: "Binary decisions force people to make a decision rather than hiding uncertainty." The practical consequence is fewer labeled examples needed to reach statistical significance, which cuts the cost of building reliable evaluators.
Excess context in LLM judge prompts degrades accuracy; judges need only the evidence relevant to the specific failure addresses what goes into the judge prompt. Their finding is that adding context beyond the minimum needed for a specific failure degrades judge accuracy. They call this context rot. The fix is to scope each judge to the evidence relevant to its failure class. A catch-all judge with full conversation history and system prompt boilerplate performs worse than a judge given only the material its criterion requires. The two findings are a matched pair: force judgment at the annotation stage, remove noise at the judging stage.
Both cards extend the Husain thread already in the library. Automated eval agents cannot detect failures that require your product's definition of good user experience (captured September 24) established that criteria shift as annotators see more examples. Binary annotation cannot fully prevent criteria drift, but it removes the overlay of middle-value noise that makes drift harder to detect.
- Hamel Husain (with Shreya Shankar), Binary pass/fail annotation produces better quality than Likert scales by forcing judgment on uncertain cases. Binary pass/fail annotation produces better quality than Likert scales by forcing judgment on uncertain cases, reducing variance and lowering sample size requirements.
- Hamel Husain (with Shreya Shankar), Excess context in LLM judge prompts degrades accuracy; judges need only the evidence relevant to the specific failure. Excess context in LLM judge prompts degrades accuracy; judges need only the evidence relevant to the specific failure being evaluated.
Managing AI mandates upward without arguing from opinion
Kevin Indig and Amanda Johnson published "Managing the AI Mandate Upward" in Growth Memo on September 21. The premise is grounded in survey data: 69% of senior executives report using AI less than one hour a week, yet many are issuing adoption mandates. The asymmetry is the problem. Teams have ground-truth evidence of what AI does and costs. Executives have general market conviction.
Executives mandate AI adoption while using it minimally themselves; documented hour tallies beat opinion when managing upward captures the communication strategy that follows from this: document AI hours, including overhead hours, and surface them directly rather than arguing about tool quality or team readiness. Documented tallies shift the conversation from opinion to data, which executives can engage with on its own terms. The prior Indig card in the library (AI tool maintenance in marketing teams creates invisible overhead that the official hours count does not capture, making net productivity gains structurally smaller than they appear, from September 7) documented that AI overhead hours are invisible on most marketing teams. This card extends the frame: the same invisibility that makes overhead hard to manage internally also makes it impossible to communicate externally. Logging hours solves both.
- Kevin Indig (with Amanda Johnson), Executives mandate AI adoption while using it minimally themselves; documented hour tallies beat opinion when managing upward. Most executives mandating AI adoption use it less than one hour a week; documented hour tallies beat opinion when managing these mandates upward.