A cognitive assistance system with augmented reality for manual repair tasks with high variability based on the digital twin

Cognitive assistance systems with augmented reality are a promising solution to the increasing complexity in all industries. Our proposed assistance system is geared towards manual repair tasks with high variability in the maintenance, repair and overhaul industry. We use the digital twin for object-specific information and a human–machine interface that uses a web application for digital work instructions and augmented reality for spatial information. Context awareness of the assistance system is achieved by using two stationary 3D cameras and a barcode scanner. Camera images and point cloud data are used for precise markerless pose estimation and body tracking.