Distributed AutoML Pipeline Search in PC/RasPi K8s Cluster

In this lightning talk, I will present an example workflow in leveraging the Kubernetes cluster of RaspberryPis to perform parallel search in finding the best AutoML pipeline in a given classification task. While many applications of RasPis are targeted for IOT usage, a K8s cluster of RasPis running Julia can be targeted to solve more complex problems and I will provide examples of the cluster performance running AutoMLPipeline applications.

There is a growing need for low-power computing devices due to their minimal thermal and energy footprint to be used in many HPC applications such as weather forecasting, ocean engineering, smarthome computing, biocomputing, AI modeling, etc. ARM-based processors such RasPis provide an attractive solution because they are cheap, versatile, and has great Linux hardware support as well as stable Julia releases. Due to its full Linux compatibility, making a K8s cluster from a bunch of RasPis become a trivial exercise as well as running Julia's cluster manager on top of K8s. This talk will provide an overview and an example walk-through how to leverage Julia+Raspis+K8s combinations to solve certain ML pipeline optimization tasks.