Calibrate your LLM judge against human labels
Paste your judge’s verdicts next to human labels. You get a plain answer on whether to trust it, chance-corrected agreement (Cohen’s κ) with an interval, and every example where they disagree.
- Computed in your browser
- Nothing is uploaded
- No account needed
Your verdicts
One row per example, as JSONL or CSV, with an id, the judge’s label and the human label. Drop a file on the field or paste.
Paste verdicts or load a sample, and the report appears here.
How it compares
Judgewright does one job: it tells you how far to trust a judge you already run. If you need to run judges, store results or label data, the other kinds of tool are the better fit, and you can bring their verdicts here.
Where your labels goStay in your browser tabOthers
- Hosted eval platforms: Uploaded to the vendor
- A notebook one-liner: Stay on your machine
- A spreadsheet: Wherever the sheet lives
- Annotation tools: Usually the tool's server
Account neededNoOthers
- Hosted eval platforms: Yes
- A notebook one-liner: No
- A spreadsheet: No
- Annotation tools: Usually
Chance-corrected agreementκ and α, side by side with raw agreementOthers
- Hosted eval platforms: Varies; raw agreement is common
- A notebook one-liner: κ, if you call it
- A spreadsheet: Only if you build the formula
- Annotation tools: Mostly human-to-human agreement
An interval on κSeeded bootstrap, reproducibleOthers
- Hosted eval platforms: Varies
- A notebook one-liner: Only if you code it
- A spreadsheet: Rarely
- Annotation tools: Rarely
Reading the disagreementsMatrix filter and side-by-side critiquesOthers
- Hosted eval platforms: Usually strong
- A notebook one-liner: Manual
- A spreadsheet: Filter and scroll
- Annotation tools: Strong: it's where labels are made
Runs the judge for youNo: you bring the verdictsOthers
- Hosted eval platforms: Yes
- A notebook one-liner: You run it
- A spreadsheet: No
- Annotation tools: No
History across runsNo: one report at a timeOthers
- Hosted eval platforms: Yes
- A notebook one-liner: Whatever you save
- A spreadsheet: Manual
- Annotation tools: Some do
Method and input format
The verdict row Judgewright reads
One JSON object per example, one example per line. Only three fields are required: an id, what the judge said, and what the humans said. Everything else — critiques, scores, extra raters — is optional and only unlocks more of the report.
{"example_id":"ex-001",
"judge_label":"pass", // what your judge emitted
"human_label":"pass", // what the human(s) said
"judge_critique":"optional", // shown side-by-side in the table
"human_critique":"optional",
"judge_score":0.93, // optional, in [0,1]
"human_labels":["pass","pass"] // optional: ≥2 raters unlock the human ceilingQuestions
What the numbers mean, where your verdicts go, and how someone else can check your result.
What does Judgewright measure?
How far you can trust an LLM judge you already run. You paste the judge's labels next to human labels for the same examples, and Judgewright reports how much of their agreement survives a chance correction (Cohen's κ and Krippendorff's α), with confidence intervals, a confusion matrix and the exact examples where they disagree. It never runs a judge and never scores a response itself.
Why isn't percent agreement enough?
Because a judge can agree with humans just by repeating the most common label. If most of your examples pass, a judge that always says pass looks excellent on raw agreement while telling you nothing. Cohen's κ subtracts the agreement expected from the label mix alone, so Judgewright shows raw agreement, the chance baseline and κ on one scale, and says how much raw agreement flatters the judge.
Do my verdicts leave my browser?
No. Parsing, every statistic and every export run in a Web Worker inside your tab. There is no upload and no account, and the page's Content Security Policy only allows connections to its own origin plus the optional sign-in provider. The repository's browser tests fail if a full analysis ever makes an outside request.
How many labelled examples do I need?
Enough for the confidence interval to be narrow enough to decide on. Judgewright prints a seeded bootstrap interval for κ; if the interval reaches zero, the data cannot rule out a judge that is no better than chance, and the verdict says so. Practitioner guides commonly suggest a frozen set of a few hundred stratified examples, but read the interval rather than a rule of thumb.
What is the difference between Cohen's kappa and Krippendorff's alpha?
Cohen's κ compares exactly two raters: your judge and the human label. Krippendorff's α treats the judge as one member of a panel, handles several human raters and missing ratings, and supports ordinal and interval data. With one human rater and nominal labels the two agree closely. When you supply several human raters, Judgewright also reports the humans-only α, the ceiling a judge cannot meaningfully exceed.
Can someone else reproduce my numbers?
Yes. The same rows, seed and replicate count produce byte-identical results. Every report carries a receipt with the formula, the inputs, the seed, the replicate count actually used and which engine computed it, so a reviewer can recompute κ from the receipt alone.
Is it free? Do I need an account?
It is free and every analysis and export works without an account. Signing in is optional and only offered when you download a report.