Explaining Face Image Quality with SHAP

Through new technologies like Stable Diffusion [Rom+22] and large language models [Ope23], society nowadays is at a point where machine learning is critically important and prevalent in scientific research. A byproduct of these developments is that humans distrust the decisions made by those models [Ips17]. Even though many of the architectures of machine learning models are inspired by the human brain, they have reached an uncanny valley in the similarity of their decisions to those made by humans. Nevertheless, machine learning has established itself in everyday life. Face recognition, the way computers match a face to an identity, is commonly implemented using machine learning. It is used for everything from unlocking a smartphone to crossing international borders. Automated border control systems utilize face image quality estimation to check if a face image is suitable for face recognition software. The resulting quality value is low if parts of the face are covered, bad lighting is used, or other similar hindrances are present in the image. However, if the quality of a face image is evaluated so low that the image is rejected, the user gets no details about the reasoning behind it. Combined with the general need for more explanations in the context of machine learning, pixel-level face image quality can solve this problem. It provides a quality map of the image to localize high and low-quality regions to convey to the user, for example, which parts of the face are obstructed or in other ways occluded from being useful for recognition. The user can then react to that by removing glasses or looking straight at the camera if it was not done before. This improves trust in the system since the decisions are made more understandable. Previous work [Ter+23a] has introduced the concept of pixel-level face image quality and one method to calculate it but has not expanded on the idea. Additionally, a metric to evaluate the accuracy of this method was only introduced afterwards [Hub+22]. The method by Terhörst et al. is not optimized under this metric. This work approaches the topic by fusing state-of-the-art face image quality assessment and an AI explainability method inspired by Shapley Values. With that, the goal is to improve upon the previously introduced method by optimizing for the metric by Huber et al. To reach that goal, three different approaches are proposed, which utilize previous work Terhörst et al. [Ter+23a], FaceQNet [Her+19; Her+20], and MagFace [Men+21] as face image quality models. All calculations are made on the Adience [EEH14] dataset and a non-public dataset focusing on varying face images of one subject. SHAP [LL17] approximates Shapley Values with the generated face image quality models. Those values are then analyzed on a quantitative and qualitative level. Additionally, the methods themselves are analyzed on their properties. This work finds that all proposed methods outperform the previous method when evaluated by the metrics of Huber et al., using the comparison scores of different face recognition systems. From that, it can be observed that using SHAP values for pixel-level face image quality calculation improves performance in general. Furthermore, it can be established from this work that SHAP on a face image quality model made with MagFace results in the highest-performing method to calculate pixel-level face image quality values.
It can also be seen that face regions are similarly prioritized throughout all used methods, including those proposed in previous work. These results show that fusing SHAP with face image quality assessment improves performance and indicates that future methods might be useful, too. For implemented systems like automated border control, this work enables accurate estimation of pixel-level quality values for feedback.