Transformer-based Self-supervised Representation Learning for Emotion Recognition Using Bio-signal Feature Fusion

In this paper, we present a new emotion recognition framework that utilizes transformer based self-supervised representations from different bio-signals and combines them into a fused representation for the task of emotion recognition. Specifically, we explore a cross-attention based fusion mechanism that can explore mutual features among different bio-signals and learn more meaningful embeddings to estimate emotions effectively. Extensive experiments on a public dataset WESAD outperform the performance of fully supervised method for emotion recognition tasks and demonstrate the benefits of self-supervised features in recognizing different emotions. We also present a series of ablation studies to validate the proposed approach.