Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Overview of the CLEF 2009 large-scale visual concept detection and annotation task

: Nowak, Stefanie; Dunker, Peter


Peters, C.:
Multimedia experiments
Berlin: Springer, 2010 (Lecture Notes in Computer Science 6242)
ISBN: 3-642-15750-5
ISBN: 978-3-642-15750-9
ISSN: 0302-9743
Cross-Language Evaluation Forum Workshop (CLEF) <10, 2009, Corfu>
Conference Paper
Fraunhofer IDMT ()
Benchmark; evaluation; image annotation; knowledge structures; concepts ontology; concept classification

The Large-Scale Visual Concept Detection and Annotation Task (LS-VCDT) in ImageCLEF 2009 aims at the detection of 53 concepts in consumer photos. These concepts are structured in an ontology which can be utilized during training and classification of the photos. The dataset consists of 18,000 Flickr photos which were manually annotated with 53 concepts. 5,000 photos were used for training and 13,000 for testing. Two evaluation paradigms have been applied, the evaluation per concept and the evaluation per photo. The evaluation per concept was performed by calculating the Equal Error Rate (EER) and the Area Under Curve (AUC). For the evaluation per photo a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and calculates a score per photo. Altogether 19 research groups participated and submitted 73 runs. For the concepts, an average AUC of 84% could be achieved, including concepts with an AUC of 95%. The classification performance for each photo ranged between 68.7% and 100% with an average score of 89.6%.