Rendering and Analysis of Faces using Multiple Images with 3D Geometry

In this paper, we present a method for the analysis and synthesis of faces that combines image-based rendering with geometry-based warping. In contrast to conventional model-based coding systems that usually rely on textured 3-D triangle mesh models for the representation of human heads, we additionally incorporate a large number of past images in order to interpolate new views. The use of real camera frames for facial synthesis leads to very natural looking results. However, the large number of variations in the face cannot all be covered by past frames. Therefore, we use image-based interpolation only to represent head turning, which usually shows most variations due to uncovered areas. The remaining motion is modeled by 3-D geometry based warping. Similarly, the tracking of facial expression and motion has to be adapted to the combined model- and image-based approach. Experiments illustrate the applicability of the algorithms for the application of virtual conferencing.