Explaining the Decisions of Convolutional and Recurrent Neural Networks

The ability to explain and understand the prediction behaviour of complex machine learning (ML) models such as deep neural networks is of great interest to developers, users and researchers. It allows them to verify the system’s decision making and gain new insights into the data and the model, including the detection of any malfunctioning. Moreover, it can also help to improve the overall training process, e.g., by removing detected biases. However, owing to the large complexity and highly nested structure of deep neural networks, it is non-trivial to obtain these interpretations for most of today’s models. This chapter describes layer-wise relevance propagation (LRP), a propagation-based explanation technique that can explain the decisions of a variety of ML models, including state-of-the-art convolutional and recurrent neural networks. As the name suggests, LRP implements a propagation mechanism that redistributes the prediction outcome from the output to the input, layer by layer through the network. Mathematically, the LRP algorithm can be embedded into the framework of deep Taylor decomposition and the propagation process can be interpreted as a succession of first-order Taylor expansions performed locally at each neuron. The result of the LRP computation is a heatmap visualizing how much each input variable (e.g., pixel) has contributed to the prediction. This chapter will discuss the algorithmic and theoretical underpinnings of LRP, apply the method to a complex model trained for the task of visual question answering (VQA) and demonstrate that it produces meaningful explanations, revealing interesting details about the model’s reasoning. We conclude the chapter by commenting on the general limitations of the current explanation techniques and interesting future directions.