Fast and stable tracking for AR fusing video and inertial sensor data

Accurate acquisition of camera position and orientation is crucial for realistic augmentations of camera images. Computer vision based tracking algorithms, using the camera itself as sensor, are known to be very accurate but also time-consuming. The integration of inertial sensor data provides a camera pose update at 100 Hz and therefore stability and robustness against rapid motion and occlusion. Using inertial measurements we obtain a precise real time augmentation with reduced camera sample rate, which makes it usable for mobile AR and See-Through applications. This paper presents a flexible run-time system, that benefits from sensor fusion using Kalman filtering for pose estimation. The camera as main sensor is aided by an inertial measurement unit (IMU). The system presented here provides an autonomous initialisation as well as a predictive tracking procedure and switches between both after successfull (re)-initialisation and tracking failure respectively. The computer vision part performs 3D model-based tracking of natural features using different approaches for yielding both, high accuracy and robustness. Results on real and synthetic sequences show how inertial measurements improve the tracking.