Explainable fatigue detection in assembly tasks through graph neural networks

Fatigue during assembly tasks can have a negative effect on subjective as well as objective quality of work. We recorded a novel dataset for the purpose of detecting fatigue in assembly scenarios. Participants were instructed to assemble and disassemble model cars with the help of a robot arm. The recordings consist of video, depth video, EEG and eye tracking data as well as questionnaires on the participants’ fatigue. The dataset can be provided to researchers on demand. In addition to recording a dataset, we implemented a proof of concept system to detect fatigue solely on image data. In our approach the eye tracking data was used to label the participants’ fatigue. Afterwards, a graph neural network was trained on poses extracted from the video data and the generated labels. The classifications of the model are made transparent through the use of explainable AI using saliency maps and GradCAM. This work can have a positive impact on human-machine interaction and assistance systems. Through explainability, we aim to increase the acceptance of such systems by workers and industries.