MicroBench Maker: Reproduce, Reuse, Improve

Benchmarking is one of the fundamental methods for analyzing the performance of computational processes or threads. In the domain of high performance computing, benchmarks are essential to assess computer systems, e.g., the TOP500 or the Green500 benchmarks are used to define the performance of machines. We address the problem of benchmarking MPI code. A common benchmarking pattern is to repetitively execute a specific code block and to take the start and the finish timestamp of each run of this code block, while iterations are logically separated using a barrier operation. Although this benchmarking scheme is very simple, it may lead to wrong conclusions, especially if the runtime of the code under investigation is very short, e.g., a reduction operation. In such scenarios, precise and re producible measurements require accurate process synchronization methods and low overhead clocks. We present a library-based approach to MPI benchmarking. Our library can be used to conduct precise measurements without having to reinvent the benchmarking wheel. This novel library is based on the ReproMPI benchmark and supports source-code annotations. An experimenter can add annotations to lines of code that should be benchmarked. This annotated code is then transformed into a benchmark code, which allows for a clear separation of concerns, as the algorithm designer can define the test scenarios while the benchmarking expert conducts experiments independently. We show the general applicability of our approach in different use cases. In one particular study, we replace all timing and synchronization code from a well-known OSU micro-benchmark with our benchmark annotations. We demonstrate that the resulting code allows to tailor the benchmark options for allowing a fine-grained performance inspection.