Detection of vehicles in multisensor data via multibranch convolutional neural networks

Convolutional neural networks, or CNNs, raised the bar for most computer vision problems and have an increasing impact in remote sensing. However, since they usually contain multiple pooling layers, detection of exact borders of small objects at their original resolution remains yet a challenging topic. Additionally, efforts are being made to reduce the amount of training data. In this paper, we investigate the potential of fully convolutional neural networks (FCNs) for individual vehicle detection in combined elevation and optical data using relatively few training samples. By the proposed multibranch CNN, we combine object recognition within a deep learning framework with the object segmentation at a high resolution, for which two CNN branches are employed. Data fusion is accomplished with a pseudo-Siamese approach. The pixelwise classification likelihood, also referred to as heatmap, is harmoniously postprocessed by a vectorization module, which is based on the minimum bounding rectangle (MBR) extraction and allows for delineation of groups of vehicles. Two methods were developed in which MBRs are supported either by pairs of parallel lines or by region growing. Our approach allows efficient training with few training samples, while delivering high-quality detection results and good computational performance. In our detailed evaluation, we investigate the benefits of data fusion and compare our approach to other state-of-the-art networks. Different datasets were used, containing optical images and elevation data, derived either from airborne laser scanning or from photogrammetric reconstruction. The obtained results are very promising with F1 scores up to 97%.