Self-Organized Neural Network Inference in Dynamic Edge Networks

Inference of large machine learning models can quickly exceed the capabilities of edge devices in terms of performance, memory or energy consumption. When offloading computations to a cloud server is not possible or feasible, for instance, due to data sovereignty concerns or latency constraints, a solution can be to distribute the inference load across multiple devices in a local edge network. We propose an approach which is capable of orchestrating multi-stage inference tasks in a mobile ad-hoc network consisting of heterogeneous devices in a self-organized and fully distributed manner. As individual edge devices may be battery-powered and volatile, the framework ensures a high degree of reliability even in dynamic environments. In particular, new nodes are automatically and seamlessly integrated into the ensemble, rendering the approach highly scalable. Moreover, resilience against spontaneous node dropouts or connection failures is implemented through adaptive task rerouting. Finally, by enabling complex inference tasks to be processed in small segments on the most suitable hardware available in the network, the ensemble is able to attain considerable pipelining performance and energy efficiency.