Rendering techniques for mixed reality

In mixed reality (MR) design review, the aesthetics of a virtual prototype is assessed by integrating a virtual model into a real-world environment and inspecting the interaction between the model and the environment (lighting, shadows and reflections) from different points of view. The visualization of the virtual model has to be as realistically as possible to provide a solid basis for this assessment and interactive rendering speed is mandatory to allow the designer to examine the scene from arbitrary positions. In this article we present a real-time rendering engine specifically tailored to the needs of MR visualization. The renderer utilizes pre-computed radiance transfer to calculate dynamic soft-shadows, high dynamic range images and image-based lighting to capture incident real-world lighting, approximate bidirectional texture functions to render materials with self-shadowing, and frame post-processing filters (bloom filter and an adaptive tone mapping operator). The proposed combination of rendering techniques provides a trade-off between rendering quality and required computing resources which enables high quality rendering in mobile MR scenarios. The resulting image fidelity is superior to radiosity-based techniques because glossy materials and dynamic environment lighting with soft-shadows are supported. Ray tracing-based techniques provide higher quality images than the proposed system, but they require a cluster of computers to achieve interactive frame rates which prevents these techniques from being used in mobile MR (especially outdoor) scenarios. The renderer was developed in the European research project IMPROVE (FP6-IST-2-004785) and is currently extended in the MAXIMUS project (FP7-ICT-1-217039) where hybrid rendering techniques which fuse PRT and ray tracing are developed.