a builder's codex
codex · release log · 2026-07-03

Agent routing before model choice, schema gaps causing retry loops, and AI search fragmentation across three non-overlapping engines

2026-07-03 · +4 insights · +1 operator

Four new cards, one new operator. Two cards land in the agent-infrastructure lane, both showing that decisions made before a model runs determine outcome more than model selection itself. One card examines what happens on the human side of an agent session when developers stop comprehending what the agent is building. One card documents Kevin Indig's new data on AI search citation fragmentation across engines.

Theme 1, Routing and schema quality determine agent outcomes before model choice matters

Tomasz Tunguz published an essay on July 1 that reframes where to focus AI agent engineering investment. Routing architecture is the first design decision in an AI agent system, not model selection captures the core design principle: the first decision in any AI agent system should be routing architecture, not model selection. A three-layer system handles this: a skill classifier converts requests into operations, a router assigns each task to a processing tier based on complexity, context size, and failure-mode signals, and a model selector picks the cheapest model meeting confidence thresholds. Tunguz documents five failure-mode signals that trigger escalation to real-time frontier inference (missing repo context, long dependency chains, risky migrations, security-sensitive prompts, high-consequence writes) and a nightly batch evaluator that updates router weights from completed sessions. Coinbase halved AI spend while token usage grew after adding routing. Across well-designed systems, 70 to 80 percent of agent tasks route to local or async models at 90 percent or more lower cost than synchronous frontier inference.

"Prioritize routing over model choice. Most AI work runs on cheap local models."

"The model choice is the last decision, not the first."

Simon Willison published case-study findings on July 2 that illustrate the same principle from the failure direction. An instruction not to re-query information the agent already has causes guessing and retry loops when the held schema is incomplete documents what happened when Datasette Agent was told not to call describe_table if it already had the information, but the schema it held contained only table names, not column names. Faced with partial information and an instruction to avoid re-fetching, the agent guessed column names, triggered query errors, and retried with variations. The fix is either including column names in the initial schema listing or softening the re-fetch prohibition. A second finding from the same research: at small eval scales (20 training questions, 90% baseline accuracy), transcript analysis reveals failure modes faster than automated prompt optimization.

"Either include column names in the prompt's schema listing or soften that advice."

Theme 2, Conceptual fluency with agent-generated code is the prerequisite for creative collaboration

Geoffrey Litt gave a talk at AIE 2026 arguing that developers who lose conceptual fluency with what an agent builds lose more than code quality visibility. They lose the capacity to initiate new creative directions. Developers need conceptual fluency with what an agent builds to remain active creative collaborators, not passive reviewers captures the claim. Checking that agent-generated code works correctly is output verification. Knowing what structures exist, what constraints apply, and what options are open is what allows a developer to think creatively about what to build next. Without that conceptual map, developers move from directing the agent to approving or rejecting its suggestions.

"You need a rich set of concepts in your mind to think creatively and fluently about how to move something forward."

Litt's mitigation techniques: have the agent quiz you on code changes it just made; build micro-worlds (small isolated environments) to explore what a changed codebase can do. Both rebuild the conceptual map after an agent session rather than verifying output correctness.

Theme 3, AI search visibility is three non-overlapping distribution systems

Kevin Indig's June 24 research summary reports findings from 3.7 million citations across 20,000 prompts covering ChatGPT, Perplexity, and Google AI Overviews. Only 2.37 percent of URLs cited by AI search engines appear across all three major platforms; 91 percent live in exactly one captures the headline number: only 2.37% of cited URLs appear in all three engines for the same prompt. Ninety-one percent appear in exactly one. AI search visibility earned in one engine does not transfer to the others. Teams measuring "AI visibility" as a single aggregate metric are averaging three separate distribution systems that barely overlap.

Indig proposes three metrics in response: Presence (how often your domain appears across engines), Portability (the fraction of citations that appear in all three engines), and Concentration (your dependence on a single engine). High Concentration is a risk signal. A single algorithm update or source-policy change can eliminate most of an organization's AI search visibility if it is concentrated in one engine.

This compounds with his May 2026 finding (High-reasoning AI mode and standard mode share only 25.6% of cited domains; AEO built for one misses three in four sources the other cites) that within any single engine, standard mode and high-reasoning mode share only 25.6% of cited domains. Engine fragmentation and reasoning-mode fragmentation are independent risks stacking on each other.

Open the full release log →