Cross-modal identification of audiovisual streams directly from the compressed domain

During the last years, a number of search and retrieval methods for audio and visual content were described in literature. Also cross-modal approaches started to emerge recently. All search methods are based on audiovisual fingerprints, which are extracted from the audiovisual data prior to the actual search. Since the most data are available in the compressed domain, they must be decompressed prior to the feature extraction. This paper describes an audiovisual search engine, which extracts their features directly from the compressed domain, without performing a decoding algorithm. The direct video feature extraction is based on motion vectors and the direct audio feature extraction is based on a polyphase matrix description. The paper depicts an overview of the search engine, which operates cross-modal on audio, and visual features and describes the evaluation of the direct feature extraction methods in detail.