Threats to validity in empirical software security research

Empirical research in secure software engineering is increasingly important to advancing the state of the art in a scientific manner [16, 17]. Several recent results have pointed to problems related to how security research is conducted or reported in a way that is not advancing the area scientifically. Science of Security (SoS) is an area of research that seeks to apply a scientific approach to the study and design of secure and trustworthy information systems [16, 17]. The core purpose of science is to develop fundamental laws that let us make accurate predictions. Currently, the only prediction we can usually make confidently in secure software engineering is that a system will eventually fail when faced with sufficiently motivated attackers. However, there is a need and an opportunity to develop fundamental research to guide the development and understand the security and robustness of the complex systems on which we depend.