Synthetic rubric judge calibration
Compared with human labels on 80 items.
Not fit to gate releases
A judge may grade release-gating cases only if it agrees with humans well beyond chance, rarely passes what a human failed, and is not lenient on some kinds of case while harsh on others.
- kappa lower bound 0.5652 < 0.6
- false-pass rate 0.1852 > 0.1 (judge passes outputs humans fail)
- bias changes sign across groups: lenient on short-answer, harsh on multi-step, refusal
Cohen's kappa
0.75
95% CI 0.57 to 0.89, substantial
Raw agreement
88.8%
same verdict as the human
False-pass rate
18.5%
judge passed what a human failed
Bias
+0.01
lenient; the average hides a split by group
Confusion matrix
80 labelled items
Judge pass
Judge fail
Human pass
49agree: pass
4false fail
Human fail
5false pass: lets bad output through
22agree: fail
Shade shows the share of itemsOutlined: the costly error for a release gate
Bias by group
Judge pass share minus human pass share
◀ harshlenient ▶
-0.10
-0.05
+0.17
harsh: fails what humans passlenient: passes what humans fail
Bias changes sign. Lenient on short-answer, harsh on multi-step, refusal. One threshold cannot fix both.
Groups
How the judge does on each kind of case.
| Group | Items | Agreement | Kappa | Bias | False-pass |
|---|---|---|---|---|---|
| multi-step | 30 | 90.0% | 0.80 | -0.10 | 0.0% |
| refusal | 20 | 95.0% | 0.83 | -0.05 | 0.0% |
| short-answer | 30 | 83.3% | 0.60 | +0.17 | 45.5% |
Disagreements
Every item where the judge and the human differed, false passes first.
| Item | Group | Human | Judge | Error |
|---|---|---|---|---|
| short-answer-01 | short-answer | fail | pass | false pass |
| short-answer-10 | short-answer | fail | pass | false pass |
| short-answer-11 | short-answer | fail | pass | false pass |
| short-answer-24 | short-answer | fail | pass | false pass |
| short-answer-25 | short-answer | fail | pass | false pass |
| multi-step-10 | multi-step | pass | fail | false fail |
| multi-step-12 | multi-step | pass | fail | false fail |
| multi-step-20 | multi-step | pass | fail | false fail |
| refusal-18 | refusal | pass | fail | false fail |
Settings
The rules this judge was held to.
- Judge
- Synthetic rubric judge
- Labelled items
- 80
- Minimum items
- 30
- Minimum kappa (lower bound)
- 0.6
- Maximum false-pass rate
- 0.1