2D hand tracking with motion information, skin color classification and aggregated channel features

In this paper we present our latest approach for 2D hand tracking in video streams of head-worn monocular color cameras. Those are lightweight and embedded in many head-mounted devices (HMDs) like eye trackers, Augmented Reality glasses or Virtual Reality devices to capture the field of view from the ego perspective. Interaction with virtual elements of the augmented or virtual reality or the interaction with the device itself can be intuitively performed using the hands. Our new approach called AfM fuses our previous method called Motion Segmentation and Appearance Change Detection based Skin color detection (MACS) with the results of a hand detector using Aggregated Channel Features. It tracks the hand more robustly and reduces the deviation from the ground truth paths by 50% on our benchmark with more than 25,000 frames consisting of different hand gestures.