A systematic evaluation approach for data stream-based applications

The ubiquitous use of mobile devices, sensors, and the linkage between them open up opportunities for new applications using near real-time data processing and analytics. Demands on information systems are high: huge amounts of data have to be processed and results have to be delivered in near real-time. These needs are tackled by the field of Data Stream Management. Processing data streams differs in many ways from static data set processing as much of the data cannot be stored persistently. Likewise, development methods for data stream-based applications have to be specifically adapted. Process modeling has proved to increase the quality of information systems, but there exists no model specifically for data stream applications. Furthermore, the production of high quality applications requires means for a structured, iterative evaluation of the application and its outcomes. Particularly, applications fed by unreliable data sources, such as sensors, are prone to quality losses and errors. Hence, measurement, monitoring, and optionally the correction of data quality problems must take a crucial part in the development and evaluation of data stream applications. Data quality management needs to be domain and application independent and smoothly integrated into a data stream management system. These requirements have not been met satisfactorily so far.We counter the aforementioned issues by three main contributions. First, we propose a process model specifically tailored to the design, implementation, and, in particular, for the evaluation of data stream applications. To this end, we contribute a thorough analysis of data stream management principles and technologies. We also analyze existing process models in information management and discuss their suitability to data stream applications. Second, we propose evaluation methodologies embedded into the process model. Along these methodologies we design and implement a flexible evaluation framework for data stream applications. Finally, we propose a methodology and framework for data quality management for data stream applications. We first analyze quality dimensions and metrics relevant to data stream applications. We elicitate existing data quality management methodologies and present a methodology for data stream-based applications. As a major contribution we implement a flexible, domain and application independent data quality management framework for relational data stream management systems based on the proposed methodology.The process model and frameworks have been developed and empirically validated and evaluated in the context of two domains, namely Connected Intelligent Transportation Systems and Mobile Health. Algorithmic solutions for particular problems in the target domains have been devised and applied. Iterative evaluations using the proposed frameworks led to crucial optimizations of the application results.