Failure rates per time for autonomous driving safety assessment from CARLA simulation

Autonomous Driving simulations like CARLA offer a variety of functionalities including the modeling of subfunctions of autonomous driving such as detection, classification, identification, and tracking of persons, objects and road elements, route planning, and maneuver planning. All of these functionalities operate at high resolution level regarding scenario details such as conditions of weather, daytime, road type, traffic conditions, etc. In contrast, analytical safety assessments like functional
Failure Mode and Effects Analysis (FMEA), Fault Tree Analysis (FTA) or Markov modelling and simulation operate at an abstract level in terms of failures per time or conditional failure rates such as failure of object detection given certain weather or daytime conditions. Furthermore, simulation options can be varied over a wide range of parameters and scenario types, including statistical generation of scenarios and active road participants, i.e. spawning of objects. The article presents a quantitative approach to generate failure rates per hour from CARLA simulation for object detection under a range of environmental conditions in terms of precipitation intensity, fog density, and time of the day, single and in combination. Focus is on the derivation of failure rate expressions that are accessible from simulation data. To this end computer vision metrics are used together with additional information available within simulation setup to compute failure rates per hour. Results are presented using sample tables, box plot and violin graphs. The CARLA simulator is used to assess an object detection algorithm that has been fine-tuned with CARLA sample images. It is discussed why the obtained failure rates are consistent but rather high. Further improvement options of the overall approach are provided