2D audio-visual localization in home environments using a particle filter

Multimodal algorithms benefit from the advantage that they can mutually compensate the weaknesses of the individual modalities. Therefore, we propose a system to localize concurrent speakers in a two dimensional (2D) space jointly using a combined audio-visual localization algorithm. The acoustic source localization is calculated by the multichannel cross-correlation coefficient (MCCC) algorithm and the visual localization is accomplished by the SHORE TM, (Sophisticated High-speed Object Recognition Engine (SHORE), Trademark of Fraunhofer IIS, 91058 Erlangen (Germany)), video localization system. The multimodal fusion is performed by a particle filter with adaptations to the particle weighting. An evaluation of the proposed algorithm in an home-environment living lab is performed focussing on possible gains obtained by the complementary localization modalities.