Virtual video conferencing using 3D model-assisted image-based rendering

In this paper, we present a method for virtual video conferencing, where multiple distant participants can meet in a virtual room. The use of a synthetic 3-D computer graphics scene allows more than two partners to join the discussions even if they are far apart from each other. Each partner is recorded by a single camera and the video objects are inserted into the artificial scene. In order to place multiple participants into a common room, viewpoint modification is necessary which requires information about 3-D structure by means of depth maps. Since hair and uncovered areas are difficult to model by pure 3-D geometry-based warping, we add image-based rendering techniques to the system. Head turning, for example, which usually shows most variations due to uncovered areas is completely modeled from previously stored frames. In order to limit the number of frames in the buffer, other movements like head roll and tilt, as well as global translation and jaw movements are warped using a three-dimensional triangle mesh. By interpolating novel views from a 3-D image volume together with a 3-D model, natural looking results can be achieved.