Adaptive Distributed SGD
A runtime controller that switches between Parameter-Server and Ring-AllReduce training based on measured latency and worker failures.
Code on GitHub · Team project for the Parallel & Distributed Computing course at NUST (three students).
Ring-AllReduce is bandwidth-efficient but stalls when one worker is slow or has died. A Parameter Server tolerates failures but costs more messages. We built a single-host simulation framework comparing Parameter Server, Ring-AllReduce, and an adaptive hybrid that switches between them at runtime.
- A finite-state controller tracks per-iteration latency with an exponential moving average and detects fail-stop workers from heartbeats. It switches modes at iteration boundaries, with hysteresis and a minimum hold time.
- Scenarios cover latency injection and node failures, including a dynamic high-latency-plus-failure case.
- The experiment matrix sweeps scenarios × configurations × worker counts × repetitions, reports bootstrap confidence intervals, and records provenance for every run. Unit tests cover the controller and communication modes.