Rethinking the evaluation methodology of authorship verification methods

Authorship verification (AV) concerns itself with the task to judge, if two or more documents have been written by the same person. Even though an increase of research activities in the last years can be observed, it can also be clearly seen that AV suffers of well-defined evaluation standards. Based on a comprehensive literature review of more than 50 research works including conference papers, journals, bachelor's/master's theses and doctoral dissertations, we could not identify consistent evaluation procedures that adequately reflect the reliability of AV methods. To counteract this, we propose an alternative evaluation methodology based on the construction of reliable corpora in combination with a more suitable performance measure. In an experimental setup our approach reveals the weakness of a number of existing and successful AV methods, in particular, when it comes to accept as many documents of the true author, while at the same time reject as many documents of other authors, as possible.