Dense 3D reconstruction and object recognition using a minimum set of inside-out images

Dense 3D Reconstruction of environments is important for various applications like augmented reality, artefact digitization and object classification. Object classification in particular allows for scene understanding. This work proposes the development of a pipeline for image based 3D reconstruction and object recognition. The 2D images under consideration are the inside out images of the interior of a room. A dense 3D reconstruction allows the description of the room as point clouds on which the object recognition algorithms are implemented. To allow for flexibility in terms of image acquisition methods, the algorithm is robust to the type of image input as well as the number of images. Matching algorithms like Scale Invariant Feature Transform (SIFT) provide accurate correspondences while Structure from Motion (SFM) algorithms use these correspondences to estimate precise camera pose and Multi-view Stereo (MVS) methods take images with pose as input and produce dense 3D models. The reconstructed scenes are then acted upon by the 3D feature extractor and the features are compared with pre-trained classifiers from a database to carry out object recognition. The pipeline has been developed to allow for different types of Multi-view Stereo input images. While plane images allow for cheap equipment, spherical and cylindrical panoramic images allow for ease in image acquisition. We have used a modified version of the existent Open Street Map (OSM) Bundler for Structure from Motion and an overlapping view clustering problem for the Multi-view Stereo. Iterative Closest Point algorithm allows for integrating the depth maps to generate the mesh model. The pipeline also allows for an input generated using the Microsoft Xbox Kinect. We have used the Cluster Viewpoint Feature Histogram algorithm (CVFH) for the object recognition and also proposed the use of Normal Aligned Radial Features (NARF). We also study the prospect of using 2D to 3D feature correlation to find objects in the 3D generated model of the room from a 2D image of that room. This work also shows the results of a comparative study undertaken between the different possible methods to complete the task. We also study the different image geometries to further explore the invariance to camera models. Finally the pipeline has been integrated in the Rapid Prototyping Environment (RPE) framework of the Department Interactive Engineering Technologies as a plug-in to provide additional functionality.