Towards Hybrid Computer Vision: Assurance of Deep Learning based Environmental Perception with traditional Computer Vision in Advanced Air Mobility

The application of autonomous unmanned aerial vehicles (UAVs) within the Advanced Air Mobility (AAM) vision has the potential to significantly contribute to the resolution of several challenges in the fields of transportation, logistics, industrial automation and space travel. In order to enable the safe and large scale deployment of autonomous UAVs, it is crucial to ensure a trustworthy perception of the environment. The complexity of environmental perception necessitates the utilization of deep learning (DL) solutions, which are characterized by their opacity and lack of traceability in decision-making processes. Given these considerations, there have already been first regulatory initiatives from the EU and EASA that require conceptual safeguards for DL based environmental perception. The Group of Highly Automated Flying at the Fraunhofer Institute for Transportation and Infrastructure Systems IVI in Ingolstadt is therefore conducting research into the development of trustworthy environmental perception as part of the VERUM research project, in which this master thesis is conceptually integrated. This thesis explores the potential of employing a framework based on traditional computer vision (CV) methods to assure the trustworthiness of DL based environmental perception for AAM. The aim of the framework is to utilize traditional CV methods, which enable the detection of matches or mismatches in the output of DL models and subsequent confidence adjustments. By avoiding these potentially untrustworthy areas, accidents could be prevented and human lives could be protected in downstream decision-making processes. In the initial conceptualization phase, a comprehensive literature survey was conducted to identify traditional CV methods as potential candidates to assure semantic and topological DL model outputs. These traditional CV methods were further analyzed to design and develop approaches to assure these DL models. The results serve as the foundation for the subsequent design and implementation of the framework "Hybrid Computer Vision".
The prototype of the developed framework was evaluated and examined to determine the extent to which trustworthy and untrustworthy areas were recognized in the environmental perception and assigned with increased or decreased confidence, respectively. The findings of this thesis demonstrate the capability to assure DL based environmental perception with traditional CV, thereby representing a first concrete solution to the conceptual regulations.