Tracking deformable surfaces with optical flow in the presence of self occlusion in monocular image sequences

In this paper, we present a direct method for deformable surface tracking in monocular image sequences. We use the optical flow constraint instead of working with distinct features. The optical flow field is regularized with a 2-dimensional mesh-based deformation model. The formulation of the deformation model contains weighted smoothing constraints defined locally on topological vertex neighborhoods. 2-dimensional deformation estimation in the presence of self-occlusion is a very challenging problem. Naturally, a 2-dimensional mesh folds in the presence of self-occlusion. We address this problem by weighting the smoothness constraints locally according to the occlusion of a region. Thereby, the mesh is forced to shrink instead of fold in occluded regions. Occlusion estimates are established from shrinking regions in the deformation mesh. Finding the best transformation then amounts to minimizing an error function that can be solved efficiently in a linear least squares sense.