Multimodal Object Interaction in Virtual Environments

This paper investigates the applicability of a multimodal metaphor for interacting with virtual objects. Strictly speaking, we want to combine speech with gestural input in terms of improving the user's articulation possibilities. Mainly, we focus on making the object selection more precise, since this is one of the major lacks of the in [3] introduced (unimodal) gesture recognition metaphor. As discussed in [3] and [4], exact selection of objects from a distance (especially when objects are grouped together) becomes more difficult the further away the user is located. Because of the naturalness of the metaphor, it might become impossible to select a specific object just by looking and pointing at it from a distance. We will define a space of ambiguity that contains objects where an exact selection is difficult, depending on the gestures' and the environment's properties. In combination with speech recognition, we are able to specify which of the contained objects is meant and support the user's intentions on a human-like basis. Beside speech and gestures, no other utilities are used, thus the extended metaphor remains natural and is well suited for human-computer interaction.