M.Sc.

Schwaiger, Adrian

Filters

4
4
Reset filters

Settings
Now showing 1 - 4 of 4
  • Publication
    Beyond Test Accuracy: The Effects of Model Compression on CNNs
    ( 2022)
    Schwaiger, Adrian
    ;
    Schwienbacher, Kristian
    ;
    Roscher, Karsten
    Model compression is widely employed to deploy convolutional neural networks on devices with limited computational resources or power limitations. For high stakes applications, such as autonomous driving, it is, however, important that compression techniques do not impair the safety of the system. In this paper, we therefore investigate the changes introduced by three compression methods - post-training quantization, global unstructured pruning, and the combination of both - that go beyond the test accuracy. To this end, we trained three image classifiers on two datasets and compared them regarding their performance on the class level and regarding their attention to different input regions. Although the deviations in test accuracy were minimal, our results show that the considered compression techniques introduce substantial changes to the models that reflect in the quality of predictions of individual classes and in the salience of input regions. While we did not observe the introduction of systematic errors or biases towards certain classes, these changes can significantly impact the failure modes of CNNs and thus are highly relevant for safety analyses. We therefore conclude that it is important to be aware of the changes caused by model compression and to already consider them in the early stages of the development process.
  • Publication
    Measuring Ensemble Diversity and its Effects on Model Robustness
    ( 2021)
    Heidemann, Lena
    ;
    Schwaiger, Adrian
    ;
    Roscher, Karsten
    Deep ensembles have been shown to perform well on a variety of tasks in terms of accuracy, uncertainty estimation, and further robustness metrics. The diversity among ensemble members is often named as the main reason for this. Due to its complex and indefinite nature, diversity can be expressed by a multitude of metrics. In this paper, we aim to explore the relation of a selection of these diversity metrics among each other, as well as their link to different measures of robustness. Specifically, we address two questions: To what extent can ensembles with the same training conditions differ in their performance and robustness? And are diversity metrics suitable for selecting members to form a more robust ensemble? To this end, we independently train 20 models for each task and compare all possible ensembles of 5 members on several robustness metrics, including the performance on corrupted images, out-of-distribution detection, and quality of uncertainty estimation. Our findings reveal that ensembles trained with the same conditions can differ significantly in their robustness, especially regarding out-of-distribution detection capabilities. Across all setups, using different datasets and model architectures, we see that, in terms of robustness metrics, choosing ensemble members based on the considered diversity metrics seldom exceeds the baseline of a selection based on the accuracy. We conclude that there is significant potential to improve the formation of robust deep ensembles and that novel and more sophisticated diversity metrics could be beneficial in that regard.
  • Publication
    Is Uncertainty Quantification in Deep Learning Sufficient for Out-of-Distribution Detection?
    ( 2020)
    Schwaiger, Adrian
    ;
    Sinhamahapatra, Poulami
    ;
    Gansloser, Jens
    ;
    Roscher, Karsten
    Reliable information about the uncertainty of predictions from deep neural networks could greatly facilitate their utilization in safety-critical applications. Current approaches for uncertainty quantification usually focus on in-distribution data, where a high uncertainty should be assigned to incorrect predictions. In contrast, we focus on out-of-distribution data where a network cannot make correct predictions and therefore should always report high uncertainty. In this paper, we compare several state-of-the-art uncertainty quantification methods for deep neural networks regarding their ability to detect novel inputs. We evaluate them on image classification tasks with regard to metrics reflecting requirements important for safety-critical applications. Our results show that a portion of out-of-distribution inputs can be detected with reasonable loss in overall accuracy. However, current uncertainty quantification approaches alone are not sufficient for an overall reliable out-of-distribution detection.
  • Publication
    Machine Learning in sicherheitskritischen Systemen
    ( 2020)
    Schwaiger, Adrian
    Der Einsatz von Machine Learning (ML), insbesondere von Deep Learning, ermöglicht erst viele hochkomplexe Anwendungen, beispielsweise in der Medizintechnik oder bei autonomen Systemen. Derzeit gibt es beim Einsatz in solchen sicherheitskritischen Systemen jedoch noch einige Herausforderungen. Drei dieser Probleme und Möglichkeiten, wie diese in Zukunft gehandhabt werden können, sollen im Nachfolgenden am Beispiel des autonomen Fahrens vorgestellt werden.