Multimodal 3D measurement setup for generating multimodal real-world data sets for AI-based transparent object recognition

Multimodal 3D imaging is a key technology in various areas, such as medical technology, trust-based human-robot collaboration and material recognition for recycling. This technology offers new possibilities, particularly for the 3D perception of optically uncooperative surfaces in the VIS and NIR spectral range, e.g., transparent or specular materials. For this purpose, a thermal 3D sensor developed by Landmann et al. allows the 3D detection of transparent and reflective surfaces without object preparation, which can be used to generate real multimodal 3D data sets for AI-based methods. The 3D perception of optically uncooperative surfaces in VIS or NIR is still nowadays an open challenge (cf. Jiang et al.). However, to overcome this challenge with AI-based networks for segmentation, pose estimation or 3D reconstruction (monocular/stereo), data sets with optically uncooperative objects are mandatory. Currently, only few real-world data sets are available. This is due to the high effort and time-consuming process of generating these data sets with ground truth. Currently, transparent objects must be prepared, e.g., painted or powdered, or an identical opaque twin of the uncooperative object is needed (for 3D reconstruction). Currently, transparent object must be labeled manually for segmentation. This makes data acquisition very time consuming and elaborate. We present our multimodal 3D measurement system as well as our new measurement principle, with which we can generate real multimodal 3D data sets with annotation without object preparation techniques. This system significantly reduces the effort required for data acquisition. We also show the advantages and disadvantages of our measurement principle and data set compared to other data sets (generated with object preparation), as well as the current limitations of our novel method. In addition, we discuss the key role of data sets in AI-based methods.