Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG

Quality assurance in general and testing in particular remains as an unsolved problem in product line engineering. It is unclear, from an economic point of view, whether one should test each product line member separately or test just product-specific parts and compose them with tested core assets from family engineering. While the former strategy introduces redundancy in testing, the latter strategy is also more difficult because of genericity. Ideally, we need to carefully calculate cost-benefits of both cases and define a test strategy accordingly. Currently, there is no economic model for testing in the context of product lines. Thus, we introduce an initial economic model, and then perform Monte-Carlo simulations to calculate cost-benefit of the two test strategies. For the Testo product line of climate and flue gas measurement devices, the results showed that, using an infrastructure-focused test strategy, on average a cost saving of 13% can be expected with 87% certainty.