Pottics - the potts topic model for semantic image segmentation

We present a novel conditional random field (CRF) for semantic segmentation that extends the common Potts model of spatial coherency with latent topics, which capture higher-order spatial relations of segment labels. Specifically, we show how recent approaches for producing sets of figure-ground segmentations can be leveraged to construct a suitable graph representation for this task. The CRF model incorporates such proposal segmentations as topics, modelling the joint occurrence or absence of object classes. The resulting model is trained using a structured large margin approach with latent variables. Experimental results on the challenging VOC'10 dataset demonstrate significant performance improvements over simpler models with less spatial structure.