Towards a Lightweight Approach for On-site Interaction Evaluation of Safety-critical Mobile Systems

The infrastructure organization of large-scale events involves high safety requirements for the visitors and is a central issue for the officials in charge. To assist in dealing with this, we developed the RESCUER app, which runs on smartphones and allows the crowd to report about an emergency, thereby improving the process for rescuing humans in an emergency. For the evaluation of the app, we faced the problem that people participating in a large event, such as a soccer match, are not willing to spend time on completing a long survey or interview. Hence, the goal of this contribution was to select and perform an on-site mobile evaluation approach that fits this context and allows us to evaluate the user interaction. The evaluation took place during the FIFA World Cup 2014 and tested the usability of the app with 112 users in Brazil and in Germany. As a result of the evaluation, we found severe usability issues and concrete insights into how to solve them. For our evaluation approach, it means that on-site mobile evaluation is an appropriate method for improving the usability and interaction of safety-critical software systems.