Material Segmentation for Visual aware Recommender Systems

People nowadays have the possibility to get recommendations for almost anything based on things they previously purchased or liked. These recommendations are often based on categories, simple colors, or other user interactions. This work presents a different approach by using precise material recognition to recommend furniture as well as clothes. These so called visual aware recommender systems are fairly unknown and have only recently gained attention. A visual aware recommender system extracts visual features from its input and uses these features to recommend accordingly. One of the biggest advantages is that these systems do not suffer from the cold start problem that many modern recommender systems have, since they do not require any other information except the visual input. In order to use material information for recommendations, precise semantic segmentation is required. Therefore, the two best performing state-of-the-art neural networks for this task are compared and evaluated, while the better model is then used in the recommender system. Performance is demonstrated by using not just one approach, but two approaches. One uses a user study to evaluate the performance gain compared to a recommender system without material recognition, and the other uses expert data known to be true to evaluate the total precision on a real live task. Both of them confirm the assumption that material recognition not only works, but also substantially improves recommendation performance especially on certain combinations.