Automatic understanding of group behavior using fuzzy temporal logic

Automatic behavior understanding refers to the generation of situation descriptions from machine perception. World models created through machine perception can be used by a reasoning engine to deduce knowledge about the observed scene. For this study, the required machine perception is annotated, allowing us to focus on the reasoning problem. The applied reasoning engine is based on fuzzy metric temporal logic and situation graph trees. It is evaluated in a case study on automatic behavior report generation for staff training purposes in crisis response control rooms. The idea is to use automatically generated reports for multimedia retrieval to increase the effectiveness of learning from recorded staff exercises. To achieve automatic report generation, various group situations are deduced from annotated person tracks, object information, and annotated information about gestures, body pose, and speech activity. The contribution of this paper consists of improvements to the existing knowledge base that models the group situations, and a quantitative evaluation using a substantial set of self-developed data and ground-truth. We also describe recent improvements to the self-developed software tools for annotating and visualizing data, ground-truth, and results.