Appoint one trusted-taste expert as the eval benevolent dictator, committees stall the loop

Claim

For LLM eval work, appoint a single person whose taste the team trusts as the benevolent dictator on what counts as a failure. A committee bogged down debating the rubric never ships an eval; one trusted taste arbiter ships and the rubric tightens through use.

Mechanism

Open coding is judgment work. A committee needs consensus on the definition of "wrong" before any traces get coded, and that conversation rarely converges before the team loses momentum. A single arbiter encodes a coherent taste and writes it down. The rubric then evolves through actual reviews, not through abstract debate. Domain experts, often the product manager, make the best arbiters because they hold both the user perspective and the product reality.

Conditions

Holds when:

• The arbiter has both domain expertise and team trust.
• The team accepts that the rubric will drift, intentionally, as more traces are reviewed.
• The arbiter is willing to be visibly wrong sometimes and revise.

Fails when:

• The arbiter has authority but not taste (political appointment).
• The team cannot live with one person's judgment and re-litigates every call.
• Multiple stakeholders have legitimate but incompatible quality bars (competing customers, regulators).

Evidence

"When you're doing this open coding, a lot of teams get bogged down in having a committee... You can appoint one person whose taste that you trust."

Hamel and Shreya prefer the product manager for the role because PMs hold both user and product context.

· Hamel Husain & Shreya Shankar on Lenny's Podcast, 2026-04-28

Signals

• Trace reviews ship weekly with one named owner.
• Rubric documents are updated by the arbiter in flight, not in committee meetings.
• The team stops debating the rubric and starts debating specific traces.

Counter-evidence

For high-stakes regulated domains, single-person arbitration may not be acceptable. Multiple-arbiter calibration (with disagreement tracked and resolved) is the safer path there.

Cross-references

• Sample 100+ traces, write one free-form note per trace, let an LLM cluster the notes, humans first, machines second, the workflow the arbiter runs
• When the decision-maker is unclear, you are it, be a force for positive momentum, Claire Hughes Johnson's structurally similar rule for cross-functional decisions