• English
  • Deutsch
  • Log In
    or
  • Research Outputs
  • Projects
  • Researchers
  • Institutes
  • Statistics
Repository logo
Fraunhofer-Gesellschaft
  1. Home
  2. Fraunhofer-Gesellschaft
  3. Konferenzschrift
  4. Benchmarking Uncertainty Estimation Methods for Deep Learning with Safety-Related Metrics
 
  • Details
  • Full
Options
2020
Konferenzbeitrag
Titel

Benchmarking Uncertainty Estimation Methods for Deep Learning with Safety-Related Metrics

Abstract
Deep neural networks generally perform very well on giving accurate predictions, but they often lack in recognizing when these predictions may be wrong. This absence of awareness regarding the reliability of given outputs is a big obstacle in deploying such models in safety-critical applications. There are certain approaches that try to address this problem by designing the models to give more reliable values for their uncertainty. However, even though the performance of these models are compared to each other in various ways, there is no thorough evaluation comparing them in a safety-critical context using metrics that are designed to describe trade-offs between performance and safe system behavior. In this paper we attempt to fill this gap by evaluating and comparing several state-of-the-art methods for estimating uncertainty for image classifcation with respect to safety-related requirements and metrics that are suitable to describe the models performance in safety-critical domains. We show the relationship of remaining error for predictions with high confidence and its impact on the performance for three common datasets. In particular, Deep Ensembles and Learned Confidence show high potential to significantly reduce the remaining error with only moderate performance penalties.
Author(s)
Henne, Maximilian
Fraunhofer-Institut für Kognitive Systeme IKS
Schwaiger, Adrian
Fraunhofer-Institut für Kognitive Systeme IKS
Roscher, Karsten
Fraunhofer-Institut für Kognitive Systeme IKS
Weiß, Gereon
Fraunhofer-Institut für Kognitive Systeme IKS
Hauptwerk
Workshop on Artificial Intelligence Safety, SafeAI 2020. Proceedings. Online resource
Project(s)
ADA-Center
Funder
Bayerisches Staatsministerium für Wirtschaft, Landesentwicklung und Energie StMWi
Konferenz
Workshop on Artificial Intelligence Safety (SafeAI) 2020
Conference on Artificial Intelligence (AAAI) 2020
File(s)
N-582723.pdf (679.92 KB)
Language
Englisch
google-scholar
IKS
Tags
  • uncertainty estimatio...

  • deep learning

  • safety metrics

  • computer vision

  • safety

  • artificial intelligen...

  • AI

  • Cookie settings
  • Imprint
  • Privacy policy
  • Api
  • Send Feedback
© 2022