Evaluating AI chat answers

Evaluate with golden questions scored by rubric, not by vibe. Run the set on every model change, judge side-by-side, and review real escalated conversations weekly.

You cannot improve what you don't judge

Build a golden set: 50-100 real questions with expected outcomes. Score answers on a rubric - correct, complete, safe, on-brand - not on how fluent they sound.

Run the golden set before every model or prompt change. A change that scores worse on correctness but sounds better is a regression wearing a nice suit.

Then the weekly loop: read every escalated conversation, find the pattern, fix the prompt or the routing. Ten hours of that per month outperforms any benchmark.