Empirical investigation of the success factors of scenario based reading

In industrial settings many success stories can be found about the effectiveness and the efficiency of software inspections. In order to optimize the inspection approach different reading techniques such as checklist-based reading and scenario-based reading have been proposed. Various experiments have been performed to evaluate which of these techniques produces better inspection results; that is finds more defects with less effort. Scenario-based reading approaches performed better then ad-hoc or checklist-based approaches in some experiments but failed to improve the checklist-based approaches in others. Thus, the success factors of scenario based reading approaches need to be further analyzed. In this report we describe the design and the results of an empirical study that evaluate the impact of the detailed descriptions provided by scenario-based approaches; that is, the impact of active guidance of the inspectors on the inspector's efficiency and effectiveness. In detail, we analyzed the following hypothesis: Hypothesis H1--Team Effectiveness: Inspection teams find more defects with active guidance than without active guidance. Hypothesis H2--Team Efficiency: Inspection teams find more defects per time with active guidance than without active guidance. In addition, to the hypothesis, we analyzed the inspector's subjective perception regarding the support provided by the PBR approach; that is the support provided by active guidance. We conducted a quasi experiment (a controlled case study) to compare a focused checklist-based reading approach and perspective-based reading (as an instantiation of an scenario based approach) with respect to the influence of active guidance provided by the PBR approach. The study was conducted as part of a practical course at the technical university of Kaiserslautern that lasted over 14 weeks. In that course, students were required to change an existing house automation system. After each development phase (requirements, design, implementation), the students had to conduct an inspection of the produced artifacts. Our study focused on the requirements inspection phase, where use cases and related scenarios were inspected. 12 students participated in the study. All students had few experience with the application domain. Also, the students had few experiences in performing systematic requirements inspections. The software used in the practical course is a reactive system for house automation that was created for and evolved within the course. The system controls a building that consists of an arbitrary number of floors and rooms that have various sensors and actuators. The system consist of three sub-systems: The graphical user interface (GUI) that offers an interface to control the system. The light control system (Light) that switches lights on and off depending on the presence of people in a room and a floor. The temperature control system (Temp) that controls the room temperature, depending on the presence of people in a room and the actual day-time. A group of 4 students was responsible for the development of each subsystem. We analyzed our hypothesis within these sub-systems. The results of this preliminary study provide tendencies that active guidance results in more effective inspections and is perceived as very helpful to support individual defect detection. In Detail the preliminary study showed that: - PBR finds between 23% and 40% more defects than CBR for the Temp and GUI subsystems; - for the subsystem Light CBR finds about 32% more defects than PBR; - PBR is significant more effective than CBR for the GUI system (p=0.08) and that the other results are not significant; - CBR is more efficient than PBR for the subsystems Temp and Light; - PBR is more efficient for the sub-system GUI; - PBR is perceived as applicable (easy to use) as CBR; - PBR is perceived as harder to understand than CBR; - PBR is perceived as more useful than CBR. The results indicate the tendency that active guidance provides a benefit to the inspectors. However, the results are only first indications and need to be further investigated in controlled experiments. Moreover, based on the results of the survey we were able to identify some important research questions that need to be addressed in the future. One of the most important is that we need to investigate the influence of document size and complexity on the effectiveness and efficiency advantage of PBR; that is, we need to find out for which documents the "overhead" of PBR pays off, and when we should rather use CBR. More generally, in further research the question needs to be addressed in which context which reading technique is best suited to optimize the inspection process.