Sorbonne University, ISIR, CNRS; Concordia University, Mila; Sorbonne University, ISIR, CNRS; Concordia University, Mila
Adel Nabli, Eugene Belilovsky, Edouard Oyallon
Decentralized Optimization for Deep Learning;Asynchronous Optimization;Distributed Training;Data-Parallel
$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning