Claim
In AI agent systems, the routing layer should be designed before selecting models. A well-built router sends 70 to 80 percent of tasks to local or async models, reducing inference costs by 90 percent or more, while frontier real-time inference handles only high-consequence requests.
Mechanism
Most agent tasks do not require real-time frontier-model reasoning. A three-layer routing system handles the classification: a skill classifier converts user requests into concrete operations; a router evaluates each task against complexity, context size, and failure-mode signals to assign a processing tier; and a model selector picks the cheapest model meeting confidence thresholds for that tier. Five failure-mode signals trigger escalation to real-time inference: missing repo context, long dependency chains, risky migrations, security-sensitive prompts, and high-consequence writes. A nightly batch evaluator updates router weights by processing completed tasks against known-good outcomes, catching new failure modes before they accumulate in production. The result is that model choice becomes the last decision, not the first.
Conditions
Holds when: agent task volume is high enough that routing overhead pays off and at least half of tasks are classifiable as simple or deferrable. Fails when: all tasks require immediate frontier-model reasoning (real-time decision support, latency-sensitive user-facing queries) or when the team lacks capacity to build and maintain a three-layer routing layer.
Evidence
Tunguz cites Coinbase: after implementing "better defaults, routing, & caching," Coinbase cut AI spend nearly in half while token usage grew. Theory Ventures built its own router with synchronous failure-mode signals and a nightly evaluation loop.
"Prioritize routing over model choice. Most AI work runs on cheap local models."
"The model choice is the last decision, not the first."
"70-80% of traffic runs on local models that cost nothing per call"
Async batch reasoning costs two orders of magnitude less than real-time synchronous inference.
Signals
- AI spend falls while token count grows after routing implementation.
- The fraction of tasks routed to real-time frontier inference stabilizes below 30 percent.
- Nightly evaluator surfaces new failure-mode patterns before they appear as production errors.
Counter-evidence
Tunguz's Coinbase example involves a large organization with a specific workload mix; the 70 to 80 percent routing rate may not generalize to products where most tasks are judgment-heavy and latency-sensitive. The three-layer architecture requires sustained engineering investment that small teams or pre-product-market-fit companies may not afford. The five failure-mode signals are Theory Ventures' classification, not a universal taxonomy.
Cross-references
- Two-tier local/cloud AI routing cuts response latency by 60% and lifts throughput 25% by keeping simple tasks on-device: the June 5, 2026 card measuring Tunguz's personal two-tier workflow, where 78% of tasks ran locally and average latency fell from 47 seconds to 19 seconds.
- Accepting two-minute latency instead of two-second latency reduces AI inference costs by 6x through spot capacity and open-model routing: the June 25, 2026 card on accepting 2-minute latency to reduce costs by 6x on background workloads.