Can you trust your LLM judge?

Put the judge's verdicts next to human labels for the same items. You'll see whether it agrees beyond chance, how often it passes what a person failed, and whether it's lenient on one kind of case and harsh on another.

Runs in your browser. Your file never leaves this page.

Drop a CSV, JSONL or JSON file

One row per item, with the human verdict and the judge's verdict.

Choose a file

What the file needs

A human verdict and a judge verdict per item. Pass/fail, yes/no, true/false or 1/0 all work. Add a group to see whether the judge's bias changes by kind of case.

id	human	judge	group optional
q1	pass	pass	short-answer
q2	fail	pass	short-answer
q3	pass	fail	multi-step

80 made-up items, built to show a judge whose bias flips by group

Check it on every change

The same check runs in CI with the open-source package, and fails the build when the judge isn't fit.

# labels.jsonl: {"id", "human", "judge", "group"}
pip install holdline
holdline calibrate --labels labels.jsonl --strict

What the numbers mean

Kappa is agreement corrected for chance. If most outputs pass, a judge that always says pass agrees often and has a kappa of 0.

False-pass rate is how often the judge passed what a person failed. In a release gate, that's the error that lets a regression ship.

Bias by group shows whether the judge passes more or less than people do on each kind of case. Opposite signs mean one threshold can't fix it.