A performance benchmarking methodology for MQTT broker implementations

The rapid growth of IoT across the globe has been significant over the past decade. As the number of connected devices increases by the order of billions year over year, the capacity and operating costs of IoT networks and associated communications software becomes crucial. The manufacturers, software developers, integrators, telco operators as well as business-end users face an increasing need of a benchmarking reference that covers performance aspects of IoT transport protocols. This paper introduces a performance benchmarking methodology as well as examples for the definition of performance tests for the MQTT protocol. The implementation work was done within the open source project IoT Testware project which is part of the Eclipse Foundation. The test suites were specified in TDL-TO and realized in TTCN-3 using the open source IDE Eclipse Titan. The test specifications are covered by the standardization activities of the ETSI working group MTS TST.