Three cards, one operator. Simon Willison's September 27 year-in-review keynote and his September 29 DevDay live blog yielded three clean observations: one cost-correction to how practitioners were evaluating models, one hardware-capability finding that shifted prior assumptions, and one public benchmark for long-task agent reliability.
What real agent deployments corrected in 2026
Two trends that circulated through 2025 and early 2026 hit empirical walls when practitioners deployed agents at scale.
The tokenmaxxing proxy for model quality collapsed when agent deployments proved that cost per outcome is the binding constraint, not capability per call. names the first wall. Tokenmaxxing, using token output count as a quality proxy, held up in single-call evaluations because more tokens correlated with more reasoning. It broke in agentic workflows. A 30-step agent task at frontier prices multiplies the per-call cost by 30. At that scale the binding constraint is not capability per call but total cost per completed task. Willison put it directly from the keynote:
"the agents are expensive"
The correction matters because evaluation frameworks built on tokenmaxxing do not predict deployment economics. Teams that benchmarked models on output length and picked high-token models found the bill surprising.
The second correction came from OpenAI's own data, reported by Willison from the DevDay 2026 keynote. OpenAI reported that agent success on 8-16 hour zero-intervention tasks tripled from 10% to 35% between January and July 2026, establishing the first publicly cited improvement curve for professional-duration autonomy. captures the first public quantified trajectory for long-task autonomy: agent success on 8-16 hour zero-intervention tasks tripled from 10% in January to 35% in July 2026. That is a real improvement. It is also a 65% failure rate in July, which grounds any bullish reading. The improvement curve is the signal; the absolute number sets expectations. Together, the two cards frame the operative agent economics question for 2026: what does success actually cost, and at what rate is the success rate improving?
- Simon Willison, The tokenmaxxing proxy for model quality collapsed when agent deployments proved that cost per outcome is the binding constraint, not capability per call.. Token count as a model quality proxy collapsed when agent deployments revealed total cost per outcome as the binding constraint.
- Simon Willison, OpenAI reported that agent success on 8-16 hour zero-intervention tasks tripled from 10% to 35% between January and July 2026, establishing the first publicly cited improvement curve for professional-duration autonomy.. OpenAI reported agent success on 8-16 hour tasks tripled from 10% to 35% (January to July 2026), the first public improvement curve for professional-duration agent autonomy.
The on-device model catches the cloud
The second finding from Willison's year-in-review sits at a different layer: hardware. By mid-2026, small open-source models running on consumer laptops achieved near-frontier performance, removing the technical necessity of cloud inference for many standard tasks. records the result of his own testing. By mid-2026, small open-source models running locally on a consumer laptop had closed the performance gap to frontier cloud models to a point that changed his prior assumptions.
The mechanism is a compound of three: better quantization (4-bit and below) shrinks a 27-billion-parameter model to 17 GB without proportionate quality loss; architecture and instruction-tuning improvements raised the ceiling at smaller scale; and distillation from frontier models transferred capabilities that previously required frontier-scale training. The test case Willison cited, Qwen 3.8 27B at 17 GB, produced results that:
"running on my laptop felt almost competitive with what was going on on the frontier"
"Almost competitive" is not equivalent. Frontier models still lead on the longest reasoning chains and highest-accuracy tasks. But for the broad middle of everyday developer tasks (coding, summarization, drafting, classification), the cloud inference necessity has narrowed. Teams designing agent cost architectures now have a credible local-model tier to route routine steps through. That was not true two years ago.
- Simon Willison, By mid-2026, small open-source models running on consumer laptops achieved near-frontier performance, removing the technical necessity of cloud inference for many standard tasks.. Efficient quantization and distillation closed the performance gap between consumer-laptop models and frontier cloud models for everyday tasks by mid-2026.