Identification of misclassified pixels in semantic segmentation with uncertainty evaluation

Classification, and in particular semantic segmentation, plays a major role in remote sensing. In remote sensing, the classes usually correspond to landcover or landuse types while the data elements are image pixels. The results are so-called semantically segmented pixels describing the content of the data for each pixel. The identification of misclassified pixels is essential to perceive the overall performance of the classification algorithm. In the case of semantic segmentation, it is typically done with ground truth labels. However, such ground truth labels are rare and mostly reserved for training only. Especially deep learning approaches are data-hungry algorithms requesting a lot of labeled examples. In this work, we explore the possibility of using Monte-Carlo dropout for the identification of model-induced misclassifications. In particular, we obtain uncertainty measures from several inferences induced by the Monte-Carlo dropout. Furthermore, we examine how Markov Random Field optimization can reduce the number of misclassifications and facilitate their identification. The extent to which uncertainties provide information about misclassifications is assessed. Our results allow detecting 51 % of the misclassifications using uncertainties. Application of Markov Random Field optimization leads to a reduction of the percentage of misclassifications while detecting 0.4 % more misclassifications as without.