Why Your LLM Judge Should Say Pass or Fail, Not 1 to 5

Channel: Hamel Husain
Published: 2026-10-09
URL: https://www.youtube.com/watch?v=xYmQuVBdQO0

LLMs aren't good at continuous scores, and nobody knows what an average of 3.2 versus 3.7 actually means. A 1 to 5 scale lets you avoid making a call, a pass or fail forces one.

Transcript

It's very important for an LLM judge that you output a binary score 99% of the time. You don't want to export like a like art scale or a you know a score of 1 to five or some kind of score because that introduces tremendous complexity. LMs are not good at continuous scores. Number one. Number two is the output is not going to be clear. When you see an average score of 3.2 versus 3.7, no one really knows what the hell that means. And it's not really actionable, honestly. Um, I found that when you try to hide behind a score, you're not really making a decision. The frame here is, is this feature good enough to ship? Yes or no? Make a decision.