a builder's codex
codex · release log · 2026-07-27

Scaffolding ceilings, relay markets, and a field experiment on what AI Overviews cost publishers

2026-07-27 · +3 insights · +1 operator

Three cards, three lanes. One about model upgrades and what they break. One about the security market that emerged from cheap AI API access. One about what happens to publisher traffic when Google answers the question before the user clicks.

Theme 1, When the scaffold becomes the ceiling

Dan Shipper and the Every team spent a week with Claude Opus 5. The model argued with their instructions. It stopped mid-task. It did not work with their existing skills or plugins. Their first diagnosis was model regression. Then they deleted every skill and plugin and rebuilt from scratch. The model performed substantially better without the constraint layer.

Scaffolding built for a less capable model constrains a stronger one; upgrading models sometimes requires deleting the scaffold, not adapting it names the mechanism: scaffolding built to compensate for a weaker model's gaps encodes those gaps as permanent constraints. A more capable model that can infer intent from sparse instructions still gets dense instructions applied. Because it is capable enough to recognize overspecification, it pushes back. The failure looks like a model problem. The fix is not to adapt the scaffolding but to delete it.

"Claude Opus 5 is brilliant in flashes and frustrating in practice—it builds strong software and grinds through bugs for hours, but its best work often requires tearing down the systems you already rely on."

This pairs with Thariq Shihipar's "codebase as spec" rewrite insight from last week, and with Dex Horthy's session-reset finding on trajectory poisoning. The structure is the same in all three: accumulated context that was useful at one level becomes an obstacle at the next. The answer in all three cases is to start fresh.

Theme 2, Two things that grew in the space cheap access created

Simon Willison's July 26 post on the relay market (Open-source load-balancing proxy software creates persistent infrastructure for LLM credential relay markets; any unprotected endpoint generates financial incentive for exploitation) describes a secondary economy most builders have not seen. Two legitimate open-source load-balancing tools, one-api and new-api, get repurposed to pool stolen API credentials. Participants source credentials through free trial abuse, card fraud, and any unprotected endpoint they can find. Buyers pay below retail. The market is concentrated in China-based operations but draws from any accessible credential.

"there's now an entire ecosystem that can profit from finding a new unprotected endpoint to exploit."

The operational takeaway is direct. A hard spending cap per user per period makes any individual credential nearly worthless to a relay market operator. Without it, any app exposure is a standing profit opportunity for someone. Caps are a security requirement. They are not optional cost control.

Ananya Sen's July 21 field experiment (AI Overviews reduce outbound clicks by 40% with no corresponding improvement in user satisfaction) lands in the same conceptual space: something built for one purpose creates effects that extend past its original scope. Google's AI Overviews are designed to give better answers. The experiment, run with roughly 1,000 participants using a custom browser extension, measured what else they do. They reduce outbound clicks to publisher pages by 40%, conditional on a search occurring. Users with and without AI Overviews visible reported identical satisfaction. The summary layer resolves intent and removes click motivation without making the experience better.

"Showing AI Overviews reduces clicks by 40% conditional on a search happening"

The second finding is precise: only 7-8% of clicks in the study originated from links inside AI Overviews themselves. The block captures intent without replacing referral traffic. Publisher sites seeing click declines despite stable ranking positions should measure AI Overview exposure on their top queries before diagnosing a ranking problem.

Open the full release log →