ELOQUENT Lab at CLEF 2026: Evaluation of Generative Language Model Quality

The ELOQUENT lab for evaluation of generative language model quality and usefulness addresses high-level quality criteria for generative language models through a set of open-ended shared tasks implemented, where possible, to minimise human effort in assessment, and with an objective to study how much the languages that the foundation model has been trained on make a difference in its responses. In this third ELOQUENT edition, the three planned tasks investigate how human-like text generated by language models can be (the Voight-Kampff task), how reliably a language model handles varied but equivalent input across languages (the Robustness and Consistency task), and if a generative language model can be used productively to generate and score topical quizzes without diverging into general knowledge acquired in foundational training (the PISA task). All tasks are continued evolved versions of previous editions’ tasks.