A Comparative Evaluation of Requirement Template Systems (Summary)

Our publication at the 31^th International Requirements Engineering Conference 2023 concerns the multiple semi-formal syntax templates for natural language requirements, that foster to reduce ambiguity while preserving readability. Yet, existing studies on their effectiveness do not allow to systematically investigate quality benefits and compare different notations. We strive for a comparative benchmark and evaluation of template systems to support practitioners in selecting template systems and enable researchers to work on pinpoint improvements and domain-specific adaptions. We conduct a comparative experiment with a control group of free-text requirements and treatment groups of their variants following different templates. We compare effects on metrics systematically derived from quality guidelines. We present a benchmark consisting of a systematically derived metric suite over seven relevant quality categories and a dataset of 1764 requirements, comprising 249 free-text forms from five projects and variants in five template systems. We evaluate effects in comparison to free text. Except for one template system, all have solely positive effects in all categories. The proposed benchmark enables the identification of the relative strengths and weaknesses of different template systems. However, overall effect sizes are relatively similar. Yet, the results show that templates can generally improve quality compared to free text.