Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

ProxSGD: Training Structured Neural Networks under Regularization and Constraints

: Yang, Yang; Yuan, Yaxiong; Chatzimichailidis, Avraam; sloun, Ruud van; Lei, Lei; Chatzinotas, Symeon

Volltext (PDF; )

International Conference on Learning Representations, ICLR 2020. Online resource : Addis Ababa, Ethiopia, April 30, 2020
Online im WWW, 2020
11 S.
International Conference on Learning Representations (ICLR) <8, 2020, Addis Ababa/Ethiopia>
Konferenzbeitrag, Elektronische Publikation
Fraunhofer ITWM ()

In this paper, we consider the problem of training structured neural networks (NN) with nonsmooth regularization (e.g. `1-norm) and constraints (e.g. interval constraints). We formulate training as a constrained nonsmooth nonconvex optimization problem, and propose a convergent proximal-type stochastic gradient descent (ProxSGD) algorithm. We show that under properly selected learning rates, with probability 1, every limit point of the sequence generated by the proposed Prox-SGD algorithm is a stationary point. Finally, to support the theoretical analysis and demonstrate the flexibility of ProxSGD, we show by extensive numerical tests how ProxSGD can be used to train either sparse or binary neural networks through an adequate selection of the regularization function and constraint set.