Label-Efficient Active Fine-Tuning for Panoptic Segmentation

Comprehensive scene understanding is a fundamental requirement for technologies such as autonomous driving and robotics. Panoptic segmentation has emerged as the standard framework for this task, as it unifies semantic and instance segmentation into a coherent scene representation. However, the practical application of this approach is limited by the high costs of data annotation. Creating pixel-perfect masks for every object in a dataset requires hundreds of hours of manual labor. While active learning offers a theoretical solution to reduce this effort, traditional active learning loops are too computationally intensive for large segmentation models and suffer from the cold start problem. This thesis addressed these challenges by proposing an Active Finetuning framework specifically for panoptic segmentation. Instead of training from scratch, the method leverages the feature representations from masktransformers
pretrained on generic datasets. These feature representations are then used to select the most informative samples from an unlabeled pool of images before any domain-specific training begins. The system eliminates the need for iterative retraining and reduces computational load by converting the active learning
cycle into a single preprocessing pipeline. A key innovation of this work is the extraction of uncertainty signals from class-agnostic mask logits, which
allows for the estimation of model confusion even without domain-specific classification heads. Experiments on Cityscapes, Mapillary Vistas, and Potsdam2D revealed that uncertainty-based sampling consistently outperforms diversity-based strategies and random selection. For example, using the Mask2Former architecture on Cityscapes, Test Time Augmentation achieved an average improvement of 1:63 Panoptic Quality over random selection. Improvements reached over +30 Panoptic Quality (PQ) points over random selection on Potsdam2D for minimal sample sizes. Specifically, multi-pass extraction methods like Test Time Augmentation and Monte- Carlo Dropout proved superior to single-pass entropy by filtering out aleatoric noise and focusing on semantic ambiguities. While diversity-based methods alone often selected uninformative samples, hybrid strategies that combine uncertainty filtering with diversity sampling proved to be highly effective. In conclusion, this thesis confirms that Active Finetuning is a viable and efficient strategy for reducing annotation costs in panoptic segmentation. The proposed pipeline enables models to achieve competitive performance while utilizing only a fraction of the data required for full training. This work provides a practical path forward for deploying panoptic segmentation in real-world applications by successfully adapting generic pretrained models to specific target domains with reduced human effort.