Computer-based facial expression analysis for assessing user experience

For the majority of the users, computers are difficult and frustrating to use. The proliferation of computers in the daily life in all sort of shapes and forms becomes a significant factor for potentially aggravating and thus degrading the users' acceptance of the technology. Traditional user observation methods, aiming at improving human-computer interaction, concentrate on monitoring users within usability laboratories. To accompany the users and follow their difficulties in daily interaction with the computer or other interaction devices would present a solution to better form a picture of the quality of the users' experience. Such strategy, though, has to rely on natural ways of assessing the users' reactions, with the risk of otherwise becoming an intrusive and undesirable feature what would be turned off. Our approach borrows from natural human social interactions where body language is so much assertive of one's appraisals and reactions. We, in particular, concentrate on facial expressions as a clue to infer the users' perceived positive and negative situations. We start by reviewing the literature on psychology and affective computing regarding users' emotions and specifically the interpretation of facial expressions. We then present a comprehensive experiment and study asserting the relation of facial expressions with the user task difficulty while performing a word processing task. Subsequently, we discuss different techniques to monitor user state, and as a result focus on machine vision as a solution for unobtrusive facial expression analysis. Consequently, further investigate computer interfaces that react to users' facial expressions as a method to provide users' assistance. A related application-oriented experiment and study is presented assessing users' reactions to that monitoring modality within the context of interacting with a virtual shopping assistant. We conclude by acknowledging that while current technology certainly limits the scope of the research and the applicability of this form of user monitoring, the promising results presented strongly motivate future use of facial expressions analysis to assess the users' experience with interactive systems.