Hier finden Sie wissenschaftliche Publikationen aus den Fraunhofer-Instituten.

Transferring Information Between Neural Networks

: Ehmann, C.; Samek, W.


Institute of Electrical and Electronics Engineers -IEEE-; IEEE Signal Processing Society:
IEEE International Conference on Acoustics, Speech, and Signal Processing 2018. Proceedings : April 15-20, 2018, Calgary Telus Convention Center, Calgary, Alberty, Canada
Piscataway, NJ: IEEE, 2018
ISBN: 978-1-5386-4658-8
ISBN: 978-1-5386-4657-1
ISBN: 978-1-5386-4659-5
International Conference on Acoustics, Speech, and Signal Processing (ICASSP) <2018, Calgary>
Fraunhofer HHI ()

This paper investigates techniques to transfer information between deep neural networks. We demonstrate that a student network, which has access to information computed by a teacher network on the training data, learns faster, can be less deep and requires less labeled examples to achieve a given performance level. For that we force the student to mimic the teacher by adding a penalty term to the student's objective. We evaluate different penalty terms: (1) mean squared error between the cost gradients, (2) the Jacobian of the pre-softmax layer, (3) its row-summed version, (4) the cost gradient differences to standard double backpropagation and (5) a targeted double backpropagation via gradient derived masks. The Jacobian method improves the accuracy proportional to the difference in training examples, in contrast to the cost gradient. If the difference in accuracy between teacher and student is large enough, we find an improvement from the Jacobian information, even if both had seen the same training data. This indicates that information transfer has a regularization effect.