Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Thinking penguin: Multimodal brain-computer interface control of a VR game

: Leeb, Robert; Lancelle, M.; Kaiser, Vera; Fellner, Dieter W.; Pfurtscheller, Gert


IEEE transactions on computational intelligence and AI in games 5 (2013), Nr.2, S.117-128
ISSN: 1943-068X
Fraunhofer Austria ()
Fraunhofer IGD ()
brain-computer interfaces (BCI); multimodality; Virtual reality (VR); computer games; Forschungsgruppe Semantic Models, Immersive Systems (SMIS)

In this paper, we describe a multimodal brain-computer interface (BCI) experiment, situated in a highly immersive CAVE. A subject sitting in the virtual environment controls the main character of a virtual reality game: a penguin that slides down a snowy mountain slope. While the subject can trigger a jump action via the BCI, additional steering with a game controller as a secondary task was tested. Our experiment profits from the game as an attractive task where the subject is motivated to get a higher score with a better BCI performance. A BCI based on the so-called brain switch was applied, which allows discrete asynchronous actions. Fourteen subjects participated, of which 50% achieved the required performance to test the penguin game. Comparing the BCI performance during the training and the game showed that a transfer of skills is possible, in spite of the changes in visual complexity and task demand. Finally and most importantly, our results showed that the use o f a secondary motor task, in our case the joystick control, did not deteriorate the BCI performance during the game. Through these findings, we conclude that our chosen approach is a suitable multimodal or hybrid BCI implementation, in which the user can even perform other tasks in parallel.