Towards extending bag-of-words-models using context features for an 2D inverted index

This paper addresses the image retrieval problem of finding images in a large dataset that contain similar scenes or objects to a given query image. Often, this task is performed with the popular Bag-of-Words (BoW)-Model which quantizes local features such as SIFT for speeding up the retrieval by using an inverted file indexing scheme. We focus on the limits of the model for very large-scale datasets since the quantization of the individual feature descriptors impairs their discriminative power. Thus, with growing datasets, the model gets increasingly distracted by irrelevant images that occasionally result in similar signatures. Our goal is to also consider neighboring features and their geometry and to condense them into a new context-feature which is meant to be quantized as well. As this new quantized context information introduces a second dimension in the BoW-Model, it supports both performance and accuracy during the retrieval step. Using the public datasets Oxford5k and Holidays, we define an appropriate framework and evaluate different ways of context feature construction, dimensionality reduction and quantization.