Evaluating the Effectiveness of Memory Safety Sanitizers

C and C++ are programming languages designed for developing high-performance applications, such as web browsers and operating systems. This performance is partly achieved by sacrificing memory safety, which introduces the risk of memory bugs - the root cause of many of today’s most severe vulnerabilities. Numerous solutions have been proposed to detect and prevent memory bugs, with the most effective employing dynamic program analysis to sanitize memory accesses. These memory safety sanitizers vary greatly in their capabilities, covering different memory regions and detecting different subsets of memory bugs. While conceptual classifications of these sanitizers exist, practical and quantitative evaluations have primarily focused on performance rather than their actual bug-finding capabilities. To bridge this gap, we present MSET, a tool for evaluating memory safety sanitizers, along with an extensive functional evaluation of the most powerful and widely used memory safety sanitizers. We systematically deconstruct memory safety bugs into distinct properties, such as the memory region, the method of memory corruption, and the type of access to the target buffer. Using this systematization, our tool generates test cases that combine small and unique code templates, covering all typical memory bugs, including various forms of buffer overflows, underflows, and use-after-frees. Our functional evaluation highlights the differences between the conceptual detection potential of sanitization techniques and the bug-finding capabilities of sanitizers with similar objectives. Furthermore, it reveals that multiple sanitizers fail to achieve their conceptual potential due to incomplete or faulty implementations. Our tool is available as open source software, enabling researchers and practitioners to test their sanitizers and uncover lost potential, conceptual shortcomings, and implementation errors.