• English
  • Deutsch
  • Log In
    Password Login
    Research Outputs
    Fundings & Projects
    Researchers
    Institutes
    Statistics
Repository logo
Fraunhofer-Gesellschaft
  1. Home
  2. Fraunhofer-Gesellschaft
  3. Konferenzschrift
  4. Benchmarking Uncertainty Estimation Methods for Deep Learning with Safety-Related Metrics
 
  • Details
  • Full
Options
2020
Conference Paper
Title

Benchmarking Uncertainty Estimation Methods for Deep Learning with Safety-Related Metrics

Abstract
Deep neural networks generally perform very well on giving accurate predictions, but they often lack in recognizing when these predictions may be wrong. This absence of awareness regarding the reliability of given outputs is a big obstacle in deploying such models in safety-critical applications. There are certain approaches that try to address this problem by designing the models to give more reliable values for their uncertainty. However, even though the performance of these models are compared to each other in various ways, there is no thorough evaluation comparing them in a safety-critical context using metrics that are designed to describe trade-offs between performance and safe system behavior. In this paper we attempt to fill this gap by evaluating and comparing several state-of-the-art methods for estimating uncertainty for image classifcation with respect to safety-related requirements and metrics that are suitable to describe the models performance in safety-critical domains. We show the relationship of remaining error for predictions with high confidence and its impact on the performance for three common datasets. In particular, Deep Ensembles and Learned Confidence show high potential to significantly reduce the remaining error with only moderate performance penalties.
Author(s)
Henne, Maximilian
Fraunhofer-Institut für Kognitive Systeme IKS  
Schwaiger, Adrian  
Fraunhofer-Institut für Kognitive Systeme IKS  
Roscher, Karsten  
Fraunhofer-Institut für Kognitive Systeme IKS  
Weiß, Gereon  
Fraunhofer-Institut für Kognitive Systeme IKS  
Mainwork
Workshop on Artificial Intelligence Safety, SafeAI 2020. Proceedings. Online resource  
Project(s)
ADA-Center
Funder
Bayerisches Staatsministerium für Wirtschaft, Landesentwicklung und Energie StMWi  
Conference
Workshop on Artificial Intelligence Safety (SafeAI) 2020  
Conference on Artificial Intelligence (AAAI) 2020  
Open Access
DOI
10.24406/publica-fhg-407174
File(s)
N-582723.pdf (679.92 KB)
Rights
CC BY 4.0: Creative Commons Attribution
Language
English
Fraunhofer-Institut für Kognitive Systeme IKS  
Keyword(s)
  • uncertainty estimation

  • deep learning

  • safety metrics

  • computer vision

  • safety

  • artificial intelligence

  • AI

  • Cookie settings
  • Imprint
  • Privacy policy
  • Api
  • Contact
© 2024