Speeding up Machine Learning Inference on Edge Devices by Improving Memory Access Patterns using Coroutines

We demonstrate a novel method of speeding up large iterative tasks such as machine learning inference. Our approach is to improve the memory access pattern, taking advantage of coroutines as a programming language feature to minimise the developer effort and reduce code complexity. We evaluate our approach using a comprehensive set of benchmarks run on three hardware platforms (one ARM and two Intel CPUs). The best observed performance boosts were 65% for scanning the nodes in a B+ tree, 34% for support vector machine inference, 12% for image pixel normalisation, and 15.5% for two dimensional convolution. Performance varied with data size, numeric type, and other factors, but overall the method is practical and can lead to significant improvements for edge computing.

N/A

2022 IEEE 25th International Conference on Computational Science and Engineering (CSE)

N/A

979-8-3503-9633-1

N/A

N/A

8