Claim
Enabling reasoning mode in Qwen3.8-27B raised accuracy on number-to-word arithmetic from 23.57% to 98.8% across a controlled experiment. The jump confirms that reasoning succeeds on sequential tasks by substituting systematic computation for direct pattern recall, which fails at scale.
Mechanism
Without reasoning, a model attempts to pattern-match the answer directly. For simple operands, that works (97% accuracy on 1-3 digit numbers). For large operands (10-13 digits), the model cannot pattern-match a number it has rarely seen converted to words, so accuracy collapses to 6.44%. Reasoning forces the model to work through the problem in discrete steps: add the digits, carry as needed, convert each component to English. The step-by-step trace succeeds even at 10-13 digits because each step is simple enough to pattern-match reliably. The accuracy jump is not incremental; it is structural. The mode switch changes which cognitive operation the model applies.
Conditions
Holds when: the task requires sequential operations where each step depends on the previous; the answer cannot be recalled from training data (rare formats, large numbers, novel combinations).
Fails when: the task is a single-step lookup well-represented in training data; reasoning mode introduces so many token generations that overthinking degrades a simpler task (see ins_willison-qwen-reasoning-overthink).
Evidence
Willison ran 5,070 test cases on a DGX Spark (local hardware) without reasoning enabled. Overall accuracy: 23.57%. Performance by number size: 97.04% for 1-3 digit operands, 6.44% for 10-13 digit operands. With reasoning enabled on a 169-case pilot: 167 correct (98.8%). The experiment was inspired by Colin Frasier's 2024 GPT-4o test.
"it got the right answer in 167 out of 169 attempts"
Signals
- Your agent's accuracy on multi-step tasks is disproportionately low compared to single-step tasks of similar difficulty
- Turning on reasoning for a task that was failing produces an outsized improvement, not just a marginal one
- Token usage per call spikes with reasoning enabled but output quality justifies the cost
Counter-evidence
Reasoning mode has a known cost: Qwen3.8-27B already tends to overthink, generating very long reasoning traces even for simple tasks (see ins_willison-qwen-reasoning-overthink from August 2026). The same mode switch that fixes arithmetic failures adds unnecessary compute to tasks that do not require sequential processing. Willison's August experiment measured this as a real cost. Reasoning is not a universal upgrade; it is the right tool for a specific task class.
Cross-references
- Reasoning models apply maximum effort to trivial tasks unless explicitly tuned to a lower effort level: Willison's August 2026 finding that Qwen3.8-27B defaults to wildly overthinking simple tasks. Both cards come from Willison's own experiments on the same model; one documents reasoning as a liability, the other as an asset, depending on task type.
- Reasoning effort levels create non-linear quality jumps in creative tasks: low and medium skip thinking entirely: reasoning effort has a nonlinear effect on creative tasks. The arithmetic finding is the analogue for structured sequential tasks.