Dynamic gestural interaction with immersive environments

This paper describes our ongoing research work on deviceless interaction using hand gesture recognition with a calibrated stereo system. Video-based interaction is one of the most intuitive kinds of Human-Computer-Interaction with Virtual-Reality applications due to the fact that users are not wired to a computer. This kind of interaction should therefore be considered as the preferred kind of interaction especially for technically unversed users. Especially, if interaction with three-dimensional environments is considered, pointing is one of the most intuitive kind of interaction used by humans. Nevertheless, using a pointing posture solely lacks of some classical interaction metaphors like moving objects in 3D space. As the most important additional gestures beside the recognition of a pointing gesture, 'grab' and 'release' gestures has been identified to enable an interactive movement of 3D objects in virtual worlds. This paper describes our video-based gesture recognition system that uses two calibrated cameras observing the user in front of a large displaying screen, identifying three different hand gestures in real time and determining 3D information like the 3D position of the user's hand or the pointing direction if performed. Different scenario applications like a virtual chess game against the computer or an industrial scenario (placing filters to an air pump system in 3D space) were developed and tested.