Immersive hybrid real-time video communication using mixed camera setups

With the increase of hybrid meetings, innovative strategies are needed to enable both local and remote participants to interact and communicate in the same way as in real meetings. In the context of a standing table scenario, we present an end-to-end real-time 3D video processing workflow and demonstrator setup. This system offers on a single display two different perspectives of a remote participant for two local participants. The system consists of a 3D capture setup and processing workflow at the remote site and a novel dual-view display on the local site. The standing table scenario requires highly realistic, high resolution rendering quality on the receiving side. Hence, low-resolution depth cameras and neural network-based depth enhancement in conjunction with 8K cameras are used to achieve high quality perspectively corrected views, allowing direct eye contact for individual users. Distinct real-time view rendering is achieved without holes to provide high-quality novel views. On the local site, a novel dual-view display is used that is capable to present two different perspectives of the remote participant at two different positions in space. The display is based on a lenticular lens that is specifically designed for our use case. The prototype setup is optimized towards low latency processing and transmission in order to fulfill the constraints given from our video communication scenario.