Theoretical foundations of deep learning

In this chapter, the authors will derive the theoretical foundations of deep neural network architectures. In contrast to shallow neural topologies, deep neural networks comprise more than one hidden layer of neurons. Even though the concept and theory has been around since many decades, efficient deep learning methods were developed in the last years and made the approach computationally tractable. This chapter will hence begin with a short review of historical and biological introduction to the topic. Then, the authors will address the mathematical model of the perceptron that still forms the basis of multilayer architectures. The authors will introduce the backpropagation algorithm as a state-of-the-art method for training of neural networks. The authors will briefly cover some popular optimization strategies that have been successfully applied and are quite relevant for radar applications that are sometimes quite different from the optical domain (e.g., scarce train ing data sets). These include stochastic gradient decent, cross-entropy (CE), regularization and the optimization of other hyperparameters. The authors will then cover convolutional neural networks (CNNs), some specific processing elements and learning methods for them. We will also look at some well-known and successfully applied architectures. The focus in this chapter lies on supervised learning for classification problems. To round up this chapter, an unsupervised method for representation learning called autoencoder (AE) is illustrated.