Title: Parallel Federated Learning over Heterogeneous Devices. Abstract: Federated Learning (FL) is a cutting-edge distributed machine learning framework that enables multiple devices to collaboratively train a shared model without exposing their own data. In the scenario of device heterogeneity, the synchronous FL suffers from latency bottleneck induced by network stragglers, which hampers the training efficiency significantly. In addition, due to the diverse structures and sizes of local models, the simple and fast averaging aggregation is not feasible anymore. Instead, complicated aggregation operation, such as knowledge distillation, is required. The time cost for complicated aggregation becomes a new bottleneck that limits the computational efficiency of FL. 
In this work, we claim that the cause root of training latency actually lies in the aggregation-then-broadcasting workflow of the server. By swapping the computational order of aggregation and broadcasting, we propose a new parallel federated learning (PFL) framework, which unlocks the edge nodes during global computation and the central server during local computation. This fully asynchronous and parallel pipeline enables handling device heterogeneity and network stragglers, allowing flexible device participation as well as achieving scalability in computation.
We theoretically prove that PFL can achieve the similar convergence rate as synchronous FL, and empirically show that our framework can tolerate both stragglers and complicated aggregation tasks, which brings $1.77\times$ to $7.32\times$ speedup.