01 Tuning distributed file systems with deep RL
Distributed file systems like Lustre expose many static parameters. Defaults leave performance unused, and tuning them by hand is slow and needs expert knowledge.
Built Magpie, a deep reinforcement learning agent that explores the parameter space, using server- and client-side metrics to learn how each setting affects throughput on benchmark workloads.
Up to 91.8% higher throughput than the default Lustre configuration.
02 Predicting Spark job runtimes
Resource allocation for Spark jobs on shared clusters is usually guessed, which wastes capacity or slows jobs down.
Built ML models that predict job runtimes to guide how many resources each job gets, and reuse what they learn across different jobs and clusters. Ran experiments on a 45-node Hadoop/Spark/Kubernetes research cluster I operated.
New jobs get right-sized clusters with few or no costly test runs, and running jobs rescale to stay on target when nodes fail.
03 ML for maintenance workshops, with Siemens
In maintenance, repair and overhaul (MRO) of turbine components, each repair decision relied on engineers' manual judgment.
Partnered with Siemens to build a Smart Expert System that gives ML-based decision support throughout the repair process, and coordinated a consortium of 21 partners.
88.5% F1 on real Siemens Energy repair data, with risky underestimates of part damage cut from 8% to 5%.