Overview and Joint Report of the Robustness and Consistency Task of the ELOQUENT 2025 Lab for Evaluating Generative Language Model Quality

Generative language models are intended to be creative and responsive to the style of the conversation they engage in. The experimental Robustness and Consistency task is designed to explore how variation between content-wise equivalent inputs influences the output of a generative language model, and in this year’s edition the task focuses on how linguistic variation makes a difference for value-oriented questions. This paper is a joint report by all participants in the task.