Claim
Likert scales (1 to 5) degrade annotation quality by encoding genuine uncertainty as a middle value. Binary pass/fail forces annotators to commit, reducing variance and lowering the sample size needed to reach meaningful evaluation results.
Mechanism
Middle values on a Likert scale (3 of 5) act as an escape hatch. Annotators who are unsure select the midpoint rather than committing to a judgment. That ambiguity enters the dataset as data, not as uncertainty. Binary judgment removes the escape hatch: each annotation is a forced choice. The resulting dataset has less noise, which means fewer labeled examples are needed to reach statistical significance, and the resulting evaluator gives fewer false impressions of confidence where the evidence is actually ambiguous.
Conditions
Holds when: the annotation task involves judgment calls where humans are genuinely uncertain on some subset of cases.
Fails when: the judgment is inherently graded (e.g., severity of a security issue, degree of fluency) and collapsing to binary loses meaningful signal. In those cases, a binary anchor combined with optional severity notation preserves forced judgment while capturing gradation.
Evidence
From Husain and Shankar's FAQ, published on hamel.dev in September 2026 as a distillation of what the two learned teaching 5,000 engineers and PMs about AI evals:
"Binary decisions force people to make a decision rather than hiding uncertainty"
They note this reduces annotation conflicts and lowers sample size requirements, which directly cuts the cost of building reliable evaluators.
Signals
- Interrater agreement improves when switching from 5-point to binary labels on the same annotation task.
- The labeled dataset requires fewer examples to train a reliable judge.
- Post-annotation review shows fewer "3 of 5" scores that annotators could not later explain.
Counter-evidence
Some evaluation tasks require ordinal distinctions that binary collapses destructively (e.g., safety severity ratings where a "2" and a "5" demand different responses). Teams doing graded harm evaluation often retain fine-grained scales for precisely this reason. The binary guidance applies most cleanly to quality-of-output annotations, not to risk or severity classifications.
Cross-references
- Automated eval agents cannot detect failures that require your product's definition of good user experience: the complementary finding that annotation criteria shift as annotators see more examples, which binary judgment cannot fully prevent.
- Teams must annotate raw LLM traces to discover failure modes before applying automated evaluation: the prior sequencing rule this annotation quality guidance sits within.
- When a team says an AI product is hard to eval, the bottleneck is product design, not eval methodology: the broader principle that eval quality reflects product clarity.