Prépublication · 2026
Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling
Eugene Agichtein, Charles L. A. Clarke et al. — Monde
Source : arXiv
Résumé
Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments. To address these issues, we propose ExecJudges, a method for generating fast, inspectable and scalable judges for automatic relevance labeling. Specifically, ExecJudges uses a frontier LLM to turn as…