Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

Eugene Agichtein, Charles L. A. Clarke et al. — Monde

Source : arXiv

Résumé

Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments. To address these issues, we propose ExecJudges, a method for generating fast, inspectable and scalable judges for automatic relevance labeling. Specifically, ExecJudges uses a frontier LLM to turn as…

Actions

Citation