Simulative Uncertainty Analysis of Camera-Based Object Detection for Autonomous Vehicles

Safety assessment is an essential prerequisite for the widespread introduction of autonomous driving systems. One trending topic in autonomous driving safety assessment is the estimation of uncertainties related to perception. The estimation of perceptual uncertainties is crucial for the reasoning about the safety of the vehicle's behavior. In this thesis, we present a simulation framework for evaluating perceptual uncertainties in object detection tasks. The proposed framework can be seen as a tool to systematically analyze uncertainties associated with an object's measurement, existence, location, and size for an autonomous driving perception system. Thereby, our work contributes to uncertainty estimation and the superior assessment of vehicle safety. The proposed framework consists of two parts. The first part contains an environment and sensor model used to represent the vehicle environment and the camera sensor. From the models, we derive object measurement uncertainties based on features derived from objects themselves or the vehicle's environment. The second part represents an object detector and is based on introspective Random Forest regression models. By matching object and scene features with predictions of the You Only Look Once (YOLO) detector, we create an introspection data set. We use the data set to train the Random Forest models. The nature of the Random Forest models allows for an interpretation of YOLO's detection behavior. We use the Random Forest models to derive uncertainties associated with the existence, location, and size of an object. The obtained prediction errors of the Random Forests are sufficiently low. The models are thus able to accurately reproduce the detection behavior of YOLO based on object and scene features. To systematically evaluate uncertainties associated with an object's measurement, existence, location, and size, we implement a sampling-based uncertainty and sensitivity analysis. We implement the sampling-based analysis techniques for a perception system modelled by the simulation framework. The uncertainty analysis shows that the measurement of an object is subject to larger uncertainties than the belief in its existence. This pattern can be partly explained by the robustness of the YOLO detector to image perturbations that reduce the quality of the measurement. In the sensitivity analysis, we confirm the high sensitivity of measurement uncertainty to Gaussian noise. On the other hand, the existence, location, and size uncertainties are highly sensitive to an object's distance.