Investigation on Automated Visual SMD-PCB Inspection based on Multimodal One-class Novelty Detection

In electronics manufacturing, the inspection of defects of electrical components on printed circuit boards (SMD-PCB) is an import part of the production chain. This process is normally implemented by automatic optical inspection (AOI) systems based on classical computer vision and multimodal imaging. Despite the highly developed image processing, misclassifications can occur due to the different, variable appearance of objects and defects and constantly emerging defect types, which can only be avoided by constant manual supervision and adaption. Therefore, a lot of manpower is needed to do this or to perform a subjective follow-up. In this paper, we present a new method using the principle of multimodal deep learning-based one-class novelty-detection to support AOIs and operators to detect defects more accurate or to determine whether something needs to be changed. By combining with a given AOI classification a powerful adaptive AOI system can be realized. To evaluate the performance of the multimodal novelty-detector, we conducted experiments with SMD-PCB-components imaged in texture and geometric modalities. Based on the idea of one-class-detection only normal data is needed to form training sets. Annotated defect data which is normally only insufficiently available, is only used in the tests. We report about some experiments in accordance with the consistence of data categories to investigate the applicability of this approach in different scenarios. Hereby we compared different state-of-the-art one-class novelty detection techniques using image data of different modalities. Besides the influence of different data fusion methods are discussed to find a good way to use this data and to show the benefits using multimodal data. Our experiments show an outstanding performance of defect detection using multimodal data based on our approach. Our best value of the widely known AUROC reaches more than 0.99 with real test data.