The need of annotation for reference image data sets

Closely related to the development process of novel image processing algorithms is always the need for a systematic training and evaluation of these methods. Thus, a predefined set of images (so-called reference images) are needed, to evaluate the performance of new methods with respect to some metric. To obtain such a metric during the evaluation process, the results of the new methods have to be compared to a valid ground truth, commonly also known as gold standard". This ground truth itself needs to be acquired and annotated systematically with respect to the image processing task to be evaluated. Hence, this work deals with the need for reference image data sets for the evaluation of new image processing methods, discusses requirements for a proper annotation for any such reference image data sets, and finally gives an example for an annotation tool for reference image data sets.