Claim
Narrow visual-generation benchmarks no longer predict where a model ranks among frontier peers. The capability that now differentiates model performance in production is reliable agentic tool-calling in long conversations.
Mechanism
Willison has run the same pelican benchmark (generate an SVG of a pelican on a bicycle) against every significant model release since 2024. The test was useful as a quick cost, quality, and spatial-reasoning probe. As of July 2026, GLM-5.2 outperforms Claude Fable 5 and GPT-5.6 on the task despite not being a comparable-tier model overall. The benchmark's correlation with model ranking has broken down.
The reason: visual SVG generation optimizes for spatial reasoning and instruction following on a closed, finite task. Frontier differentiation has moved to a different capability class. Long multi-turn agentic conversations require reliable tool selection, error recovery, and context management across many steps. These don't show up in a single-prompt SVG task. A model can produce excellent pelicans while failing on the multi-step tool-calling sequences that production agents actually run.
The pelican test retains diagnostic value as a reproducible probe for cost estimation, SVG validity, and spatial reasoning. It has lost value as a comparative ranking signal.
Conditions
Holds when: the task being evaluated is agentic, multi-step, and tool-dependent. Single-task benchmarks that optimize for a narrow capability fail to predict performance across a different capability class.
Fails when: the task being evaluated is genuinely a single-turn generation task. For creative writing, image generation, or short-form answers, single-task benchmarks may still correlate with ranking.
Evidence
Willison's July 16 post on Kimi K3 (Moonshot AI, 2.8T parameters) documents the observation directly. GLM-5.2 outperformed Claude Fable 5 on the pelican task. Willison identifies the gap: the benchmark misses agentic tool-calling reliability, which he calls "the thing that matters most for today's model."
Cost data from the Kimi K3 run: $0.94 per pelican task (vs. Opus 4.8 at $1.80). 13,241 reasoning tokens for a 3,417-token response. The cost differential demonstrates the practical value of the cost-estimation use of the test, even as the ranking value has declined.
Signals
- A lower-tier model outperforms a frontier model on your benchmark.
- Benchmark rankings diverge from production user preference or task success rates.
- The capability the benchmark measures is not the capability that drives your agent failures.
Counter-evidence
Willison's observation is one practitioner's reading of one benchmark across one task type. The decoupling may be specific to SVG generation; other visual benchmarks that more closely proxy agentic tool selection may still correlate with rankings. The claim that agentic tool-calling reliability is the primary differentiator is stated without quantitative evidence specific to this post; it is an informed judgment, not a measured result.
Cross-references
- Chaining a honeypot site's nested links through web_fetch exfiltrates private user data letter-by-letter, bypassing injection safeguards that only block direct URL attacks: a case study where the agentic tool-calling surface (web_fetch link following) is precisely the capability that benchmark-only model evaluation would miss.
- Telling a frontier model to self-route tasks to cheaper subagent models outperforms pre-written routing rules on both cost and quality: Willison's earlier observation about models routing themselves to different reasoning modes, which is also invisible to single-task benchmarks.