JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Improving LLM evaluation accuracy by preserving distributional uncertainty in judgment models.
When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses t...
ArXiv CS.CL
Source
Generated insight by OnAirToday AI
JudgeMoE addresses the critical problem of information loss in LLM evaluation by leveraging full probability distributions rather than simplified scalar scores. This advancement enables more nuanced and reliable automated benchmarking by capturing the inherent uncertainty and expert disagreement present in language model outputs.

Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.