Fraunhofer-Gesellschaft

Publica

Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Characteristics of Monte Carlo Dropout in Wide Neural Networks

 
: Sicking, Joachim; Akila, Maram; Wirtz, Tim; Houben, Sebastian; Fischer, Asja

:
Volltext (PDF; )

Online im WWW, 2020, arXiv:2007.05434, 14 S.
International Conference on Machine Learning (ICML) <37, 2020, Online>
Workshop for Uncertainty and Robustness in Deep Learning <2020, Online>
Bundesministerium fur Wirtschaft und Energie BMWi (Deutschland)
VDA Leitinitiative autonomes und vernetztes Fahren; 19A19005X
KI Absicherung - Safe AI for Automated Driving
Englisch
Bericht, Elektronische Publikation
Fraunhofer IAIS ()

Abstract
Monte Carlo (MC) dropout is one of the state-of-the-art approaches for uncertainty estimation in neural networks (NNs). It has been interpreted as approximately performing Bayesian inference. Based on previous work on the approximation of Gaussian processes by wide and deep neural networks with random weights, we study the limiting distribution of wide untrained NNs under dropout more rigorously and prove that they as well converge to Gaussian processes for fixed sets of weights and biases. We sketch an argument that this property might also hold for infinitely wide feed-forward networks that are trained with (full-batch) gradient descent. The theory is contrasted by an empirical analysis in which we find correlations and non-Gaussian behaviour for the pre-activations of finite width NNs. We therefore investigate how (strongly) correlated pre-activations can induce non-Gaussian behavior in NNs with strongly correlated weights.

: http://publica.fraunhofer.de/dokumente/N-615732.html