3D vs. 2D: On the importance of registration for hallucinating faces under unconstrained poses

Face Hallucination (FH) differs from generic single-image super-resolution (SR) algorithms in its specific domain of application. By exploiting the common structures of human faces, magnification of lower resolution images can be achieved. Despite the growing interest in recent years, considerably less attention is paid to a crucial step in FH -- registration of facial images. In this work, registration techniques employed in the literature are first summarized and the importance of using well-aligned training and test images is demonstrated. A novel method to inversely map the high-resolution (HR) 3D training texture to the low-resolution (LR) 2D test image in arbitrary poses is then presented, which prevents information loss in LR images and is thus beneficial to SR. The effectiveness of our 3D approach is evaluated on the Multi-PIE and the PUT face databases. Superior qualitative and quantitative FH results to the state-of-the-art methods in all tested poses prove the necessity of accurate registration in FH. The merit of 3D FH in generating super-resolved frontal faces is also verified, revealing 30% improvement in face recognition over the 2D approach under 30° of yaw rotation on the Multi-PIE dataset.