Bio
Shreya Shankar. Co-author of the LLM-evaluation methodology captured in the corpus alongside Hamel Husain. The work treats evals as error analysis, argues for single binary LLM-as-judge rubrics over Likert scales, and uses open-then-axial coding to turn real failures into the categories a judge should measure.
Operating themes
- Evals as data analysis. Look at real outputs and code what went wrong before building any automated judge.
- Binary judges over scores. A judge answers one yes/no question well; a 1-to-5 score hides the failure mode.