Efficacy of Foundation-Model-Based Distillation for Image Classification

In this work, we explore the efficacy of knowledge distillation in the field of computer vision classification tasks. We focus on the distillation of a large foundation teacher model to a less complex student model. The teacher model, characterized by its size and complexity, trained on large amounts of data, serves as a rich source of knowledge for training a compact student model. Our methodology involves selection and comparative analysis of distillation techniques and their impact on student performance. We provide insights regarding the effectiveness of foundation model distillation techniques for improving the performance of student models in image classification. To evaluate the distillation performance, we conduct experiments primarily on the CIFAR-100 dataset.
We find that distillation methods that leverage richer teacher outputs (e.g., logits or feature-based representations) can significantly enhance the student’s performance. Additionally, we demonstrate that under specific conditions, label refinement and pre-processing techniques can further boost distillation performance. Our findings also emphasize the trade-offs between model complexity, training time, and distillation effectiveness, offering practical recommendations for distillation applications. These insights underscore the potential of foundation model distillation techniques to enhance student model performance in real-world image classification tasks.