Augmented reality in a remote tower environment based on VS/IR fusion and optical tracking

Over the past years, several augmented reality features have been developed to make Remote Tower Operations more cost-efficient and user-friendly. In the context of a national research project (The paper reports results gained in the project ""INVIDEON"" (FKZ 20 V1505A) sponsored by the Luftfahrtforschungsprogramm (LuFo) of the Federal Ministry of Transport and Digital Infrastructure Germany.), augmented reality based on visual spectrum (VS) and infrared (IR) fusion and as well as on optical tracking is a study objective. Having both VS and IR information available at any time is expected to enable more efficient air traffic control, even at restricted visibility conditions. Integrating VS and IR in one video panorama should also decrease head-down times and therefore increase situation awareness and reduce workload. The integration of two different sensors will be realized by overlaying VS/IR combined with adapted input devices and optical tracking methods. Developing a good concept for the integration of VS/IR and testing it in an exploratory manner can only be achieved with the help of system experts and rapid prototyping methods in simulation environments. During three workshops, human factors specialists, project partners and seven air traffic controllers worked out a prototype that was gradually improved over time and helped to generate a first concept. Firstly, this paper addresses the challenges of VS/IR fusion, manual PTZ following (as a precursor for optical tracking) and adapted input devices. Secondly, it presents the construction process of a prototype in an explorative manner, based on a user-centered approach and implemented in a simulation environment. Finally, it summarizes and presents the results from the workshops and throughout the construction process.