Measuring the Effect of Background on Classification and Feature Importance in Deep Learning for AV Perception

Common approaches to explainable AI (XAI) for deep learning focus on analyzing the importance of input features on the classification task in a given model: saliency methods like SHAP and GradCAM are used to measure the impact of spatial regions of the input image on the classification result. Combined with ground truth information about the location of the object in the input image (e.g., a binary mask), it is determined whether object pixels had a high impact on the classification result, or whether the classification focused on background pixels. The former is considered to be a sign of a healthy classifier, whereas the latter is assumed to suggest overfitting on spurious correlations. A major challenge, however, is that these intuitive interpretations are difficult to test quantitatively, and hence the output of such explanations lacks an explanation itself. One particular reason is that correlations in real-world data are difficult to avoid, and whether they are spurious or legitimate is debatable. Synthetic data in turn can facilitate to actively enable or disable correlations where desired but often lack a sufficient quantification of realism and stochastic properties. To shed light on this issue and test whether feature importancebased XAI reliably distinguishes between true learning and problematic overfitting, we utilize the task of traffic sign recognition. Based on the synthesis pipeline of the Synset Signset Germany dataset, which demonstrated comparability to real-world data, we show how systematically generated synthetic data can test assumptions about feature importance-based XAI and isolate factors between classification quality and XAI values. Therefore, we systematically generate six synthetic datasets for the task of traffic sign recognition, which differ only in their degree of camera variation and background correlation. The generated datasets, which we provide for download under a CC-BY license, enable us to quantify the isolated influence of background correlation, different levels of camera variation, and considered traffic sign shapes on the classification performance, as well as background feature importance. A study of this kind is nearly impossible to conduct with real-world data, as real-world data can only be collected with difficulty at this level of comparability and without additional influencing factors. Results include a quantification of when and how much background features gain importance to support the classification task based on changes in the training domain, and show that such metrics can be indicative of complex properties of the training data and task, not purely of learning quality. Download: synset.de/datasets/synset-signset-ger/background-effect