View-Consistent Virtual Try-on of Glasses using a Hybrid NeRF-Mesh Rendering Approach

In recent times, an increasing fraction of global purchases is conducted via the world wide web. For individual accessories, such as glasses, a purchase commonly involves trying on multiple products to fit individual aesthetic preferences. The experience of the try-on process differs greatly between online and offline shopping. While there are real-time methods that facilitate virtual try-on of glasses, they usually project them onto a 2D image. This leads to inconsistent positioning of the glasses model between different views, negatively influencing the shopping experience. We propose a strategy, that enables the virtual try-on of glasses using a Neural Radiance Field as head avatar and a meshed glasses model, leading to consistent positioning of the spectacle frame through multiple views while maintaining real world like visual quality. We contribute an approach for placing and aligning the glasses in relation to the human head in the given NeRF context. Furthermore, we propose a framework for realtime hybrid rendering of meshes and Neural Radiance Fields in the same scene. The proposed method requires training times around one minute and produces a freely explorable 3D model that achieves interactive framerates on end-consumer hardware.