High quality 360° Video Rendering and Streaming on the Web
The Fraunhofer FOKUS business unit Future Applications and Media (FAME) hosts all the technical know-how needed for the realization of interactive media applications: services that can be converged on multiple media platforms, and personalizable services that enable interaction with a broad array of end devices. In all their work, the experts at FAME make consistent use of Web-based technologies. Moreover, they also play active roles in standardization committees (including ETSI, HbbTV, DASH-IF and W3C), support customers in the development of interoperable solutions, evaluate them and engineer showcases for visualization of their possible functionalities. 360° video is one of the technologies the FAME team is working on. Our primary interest is to bring 360° video experience to large screens like TVs, set-top-boxes and streaming devices (Chromecast, Android TV, EL). Our activity is driven by the huge interest from content providers especially from broadcasters to bring 360° video experience to TV. HbbTV is one of the options broadcasters can use to offer this kind of services together with the broadcast service. Our 360° Video Cloud Streaming solution enables high quality 360° video experience on low capability devices, such as Hybrid TVs (HbbTV), or in cases of constrained network connectivity e.g. on mobile devices. In 360° video the full spherical image of any direction of view is available in every moment while the spectator can freely change her individual perspective of view. The delivery of this huge source video material consumes a large bandwidth and results in a considerable processing load for the view projection and rendering. Devices as HbbTV Terminals are not capable, in terms of programmatic features, of performing the necessary image transformations. The Fraunhofer FOKUS 360° Video Cloud Streaming performs the rendering of the individual view on server side so that only the selected video content is streamed to the end device. This reduces the bandwidth needed or allows for a higher quality video delivered on a given bandwidth. It also diminishes the requirements for the end device to simply play back a usual video stream in order to provide the full 360° video experience. our solution allows content providers and broadcasters to provide an innovative video experience on traditional TV screens. Viewers can experience video content with freely selectable views on their primary video viewing device using the TV remote control. This solution is not only applicable for HbbTV terminals, but also for any device that runs a web browser like desktop, mobile and streaming devices. By using existing W3C APIs like Media Source Extension, it is possible to implement the entire logic of the 360° player in the browser which is not possible in HbbTV due the missing MSE API. In a proof-of-concept implementation we were able to play 360° video in any browser that supports MSE. The origin 360° video is prepared in a way so that there is no need for any calculation at playback time. There is also no need to stream the whole 360° video to the client but only the video content of the current view. The prepared pre-rendered video files contain segments for different combinations of view angles and video qualities. A tool that generates the pre-rendered video files is already available. Existing content delivery networks can be used to host and stream the pre-rendered videos without any additional requirement comparing to traditional video streaming. With all the advantages of this solution, there are some limitations especially regarding motion-to-photon-latency which depends on different factors like network latency, buffering strategies and segment duration. The 20ms maximum latency in case of head-mounted displays is difficult to achieve. This is why this solution is for now only suitable for devices that supports keyboard and remote control inputs. We expect from the workshop to discuss the classification of the different kind of 360° playback devices and identify functional and non-functional requirements for each device class. We are also interested to discuss potential extensions of the MSE API and HTMLVideoElement in order to provide better 360° experience.