Patch-based facial texture super-resolution by fitting 3D face models

Incorporating 3D models for face hallucination (FH) is an ill-posed problem in light of the low-resolution (LR) conditions. To deal with the challenges, a specific 3D shape modeling approach targeting LR face images is first proposed. Based on a few automatically detected 2D facial feature points, an adaptive fitting scheme to relax the fixed correspondence assumption on the facial contour is devised, allowing for pose-invariant shape recovery. In order to exploit the obtained 3D shape and pose for FH, a resolution-aware approach for registering the training 3D faces with the LR input is designed to avoid warping the LR face. Finally, the LR image formation process is reformulated to facilitate super-resolution (SR) of the 3D facial texture. Using this interpretation, the Lucas-Kanade algorithm is extended for 3D deformable models to rectify the imperfect landmark-based fitting on LR images in a posterior fashion. In this way, the final patch-wise SR stage is able to produce realistic facial textures and synthesize self-occluded regions for non-frontal poses. All in all, our main contribution is a novel and pragmatic ""2D landmarks RT 3D dense shape RT LR fitting refinement RT 3D FH"" pipeline dedicated for the LR scenario. To justify its advantages, extensive evaluation is conducted on several publicly available datasets, revealing superior accuracy over state-of-the-art methods in 3D fitting and high-quality SR results for in-the-wild faces with an interocular distance of as few as five pixels. Moreover, the frontalized high-resolution texture is also verified to help boost the performance of cross-pose face recognition.