Feature extractor sensitivity in synthetic medical image evaluation

The evaluation of synthetic images has become a crucial aspect of research, especially with the rapid advancement of generative models. Recent methodologies have introduced various metrics that not only quantify the quality of synthetic images but also assess their realism, the extent to which they cover the source distribution, and whether the models produce duplicated samples from the original datasets. This is particularly significant in medical imaging, where the utility of synthetic images hinges on their realism and proximity to the source distribution. Traditionally, synthetic images are evaluated using lower-dimensional feature spaces derived from intermediate layers of trained neural networks. The Fréchet Inception Distance (FID) remains the dominant metric, utilizing activations from an InceptionV3 model trained on the ImageNet dataset. In addition to FID, various other metrics have been proposed to address its limitations, yet most remain reliant on features from trained models. Despite its prevalence, FID's alignment with human evaluations has been questioned, especially in medical imaging contexts where the applicability of features learned from natural images is debated. The introduction of models trained on the RadImageNet dataset offers a promising alternative for extracting domain-specific features in medical imaging. This work builds on previous investigations by translating these approaches to the medical imaging domain and examining the sensitivity of different models trained on natural and medical image datasets to different augmentations. We propose to use systematic test-time data augmentations altering the evaluation data in controlled dimensions (contrast, shape) to characterize which of the pretrained models reacts in which way to the alterations. Our findings show which image features are captured by distinct representation spaces of these models, thereby facilitating the selection of models and advancing the evaluation of synthetic medical images.