Towards Quantifying Simulated Image Sensor Data: A Survey and Discussion on GAN Evaluation Metrics

Simulations are capable of solving many of today’s problems with collecting real-world data for training and testing machine learning (ML) approaches for mobility applications. However, to effectively utilize synthetic data, it must be ensured that they possess the necessary quality, meaning they are "similar enough" to real-world data and include all characteristics that ML approaches require to learn task-relevant features. As the quantitative quality assessment of image data is a difficult task, the quality of simulated images is in practice mostly determined through cross-dataset tests or performance comparisons after the addition of synthetic data. This has the disadvantage that comparable annotated real-world data are still required, which can only be obtained with significant effort or are rarely available in some domains. In recent years, research in the field of developing metrics to quantify the image quality produced by generative neural networks has made notable progress. But in general, these metrics are not trivially transferable to simulated images, as the desired metric properties diverge between Generative Adversarial Networks (GANs) and simulations. Therefore, this work analyses the differences in requirements and discusses the suitability of established GAN performance metrics for quantifying simulation based synthetic image sensor data on a theoretical basis. Thereby, a survey on existing GAN evaluation metrics is included.