Compact and Computationally Efficient Representation of Deep Neural Networks
 IEEE transactions on neural networks and learning systems 31 (2020), No.3, pp.772785 ISSN: 2162237X 

 English 
 Journal Article 
 Fraunhofer HHI () 
Abstract
At the core of any inference procedure, deep neural networks are dot product operations, which are the component that requires the highest computational resources. For instance, deep neural networks, such as VGG16, require up to 15G operations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and thus limits their use in resourcelimited environments, e.g., on embedded devices or smartphones. One common approach to reduce the complexity of the inference is to prune and quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy values are low, as measured relative to the empirical probability mass distribution of its elements. In order to efficiently exploit such matrices, one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements; therefore, cannot efficiently represent the entire set of matrices that exhibit lowentropy statistics (thus, the entire set of compressed neural network weight matrices). In this paper, we address this issue and present new efficient representations for matrices with lowentropy statistics. Alike sparse matrix data structures, these formats exploit the statistical properties of the data in order to reduce the size and execution complexity. Moreover, we show that the proposed data structures can not only be regarded as a generalization of sparse formats but are also more energy and time efficient under practically relevant assumptions. Finally, we test the storage requirements and execution performance of the proposed formats on compressed neural networks and compare them to dense and sparse representations. We experimentally show that we are able to attain up to ×42 compression ratios, ×5 speed ups, and ×90 energy savings when we lossless convert the stateoftheart networks, such as AlexNet, VGG16, ResNet152, and DenseNet, into the new data structures and benchmark their respective dot product.