SoK: Automated Software Testing for TLS Libraries

Reusable software components, typically integrated as libraries, are a central paradigm of modern software development. By incorporating a library into their software, developers trust in its quality and its correct and complete implementation. Since errors in a library affect all applications using it, there is a need for quality assurance tools such as automated testing that can be used by library and application developers to verify the functionality. In the past decade, many different systems have been published that focus on the automated analysis of TLS implementations for finding bugs and security vulnerabilities. However, all of these systems focus only on few TLS components and lack a common analysis scenario and inter-approach comparisons. Especially, the amount of manual effort required across the whole analysis process to obtain the root cause of an error is often ignored. In this paper, we survey and categorize literature on automated testing approaches for TLS libraries. The results reveal a heterogeneous landscape of approaches with a trade-off between the manual effort required for setup and for result interpretation, along with major deficits in the considered performance metrics. These imply important future directions to advance the current state of protocol test automation.