Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Combining eye tracking and physiology for detection of emotion and workload

: Brouwer, Anne-Marie; Stuldreher, Ivo; Huertas Penen, S.; Lingelbach, Katharina; Vukelic, Mathias

Fulltext ( - Gesamter Tagungsband)

Spink, Andrew (Ed.):
Measuring Behavior 2020-21. 12th International Conference on Methods and Techniques in Behavioral Research and 6th Seminar on Behavioral Methods. Proceedings. Vol.1 : 13-15 October 2021, Kraków, Poland
Kraków, 2020
ISBN: 978-90-74821-93-3
International Conference on Methods and Techniques in Behavioral Research <12, 2021, Krakow>
Seminar on Behavioral Methods <6, 2021, Krakow>
United States Air Force, Office of Scientific Research
Conference Paper, Electronic Publication
Fraunhofer IAO ()

Peripheral physiological measures such as electrodermal activity (EDA), heart rate and pupil dilation, as well as neurophysiological measures such as electroencephalography (EEG), can inform us about individuals’ cognitive and emotional state. We are interested in exploiting such measures in real life situations. A challenge of interpreting physiological measures as markers of mental state in real life is the lack of context information. We here approach this challenge by relating physiological measures to eye tracking. Participants scanned stimuli that induced different levels of workload (small sets of numbers that needed to be added or not) and different types of emotion (neutral, pleasant and unpleasant pictures). EDA, heart rate, pupil size and EEG were related to the first eye fixation on the stimulus. For peripheral measures, response traces across the following 10s were determined and signal amplitudes were compared between the different types of stimuli. EEG signals were compared for the different types of stimuli in the time interval from fixation onset to 1500 ms later using a cluster-based, nonparametric randomization approach. For the peripheral measures, high workload stimuli stood out from all other stimuli in all modalities, with patterns as expected from literature under more traditional experimental conditions: high values of EDA, heart rate, and pupil size for high compared to low workload stimuli. For emotional stimuli, peripheral physiological effects tended to be in the expected direction but were more modest in size. In the EEG signals, a significant late parieto-occipital cluster could be identified with higher amplitudes for high compared to low workload stimuli, as well as for emotional stimuli compared to the neutral stimuli. In future analyses we will combine fixation-locked signals from different modalities to detect mental states elicited by information that is being looked at. Our first results indicate that this may be especially helpful in situations related to cognitive workload, e.g. determining whether operators are not only looking at, but are also cognitively processing information that is presented on a screen.