Multispectral matching using conditional generative appearance modeling

The precise determination of correspondences between pairs of images is still a fundamental building block of many computer vision systems. Despite the maturity of modern feature matchers, multispectral methods are still lacking robustness and speed. We focus on the problem of finding point correspondences in a multispectral imaging setup. Most methods aim at invariant feature transforms (e.g. multi-modal descriptors) which come at the cost of reduced discriminance. We model the appearance change by learning an image transformation, which maps one image modality to the respective target image, conditioned on the data of the original spectral band. This approach is coupled with a pipeline of state of the art matching methods with view synthesis of increasing complexity and algorithm run-time. We evaluate the approach on a wide spectrum of multispectral datasets including near-infrared, color-infrared and night and day thermal infrared imagery. The proposed approach provides significant improvements in terms of speed and robustness compared to standard multi-modal registration approaches. In addition, the approach fits very well into existing system approaches by design. Applications are numerous and include multispectral sensor fusion, multispectral odometry systems, multispectral segmentation or multispectral super-resolution methods.