Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
- What changed
- LLM judges exhibit substantial error correlation, reducing ten judges to 3.5 independent units.
- Why you should care
- Shared training causes LLM judges to make identical mistakes.
- Your move
- Reconsider. Consensus may overstate reliability due to correlated errors.
- Evidence
- Confirmedarxiv.org ↗
- What to watch next
- Adoption of shared-error correction methods in standard LLM evaluation benchmarks.
- Event
- research
- Event date
- Sep 23, 2026
- Relevant to
- General AI readers