Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses 32x less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines. The most powerful supercomputer in the world is currently a cluster of over 27,000 GPUs at Oak Ridge National Labs (TOP500, 2018). Distributed algorithms designed for such large-scale systems typically involve both computation and communication: worker nodes compute intermediate results locally, before sharing them with their peers. When devising new machine learning algorithms for distribution over networks of thousands of workers, we posit the following desiderata: D1 fast algorithmic convergence; D2 good generalisation performance; We have analysed the theoretical and empirical properties of a very simple algorithm for distributed, stochastic optimisation. We have shown that SIGNSGD with majority vote aggregation is robust and communication efficient, whilst its per-iteration convergence rate is competitive with SGD for training large-scale convolutional neural nets on image datasets. We believe that it is important to understand this simple algorithm before going on to devise more complex learning algorithms.An important takeaway from our theory is that mini-batch SIGNSGD should converge if the gradient noise is Gaussian. This means that the performance of SIGNSGD may be improved by increasing the per-worker mini-batch size, since this should make the noise 'more Gaussian' according to the Central Limit Theorem.We will now give some possible directions for future work. Our implementation of majority vote may be further optimised by breaking up the parameter server and distributing it across machines. This would prevent a single machine from becoming a communication bottleneck as in our experiments. Though our framework speeds up Imagenet training, we still have a test set gap. Future work could attempt to devise new regularisation schemes for signed updates to close this gap. Promising future work could also explore the link between SIGNSGD and model compression.
Workers send gradient signs to the server, and the update is decided by majority vote. We show that this algorithm is convergent, communication efficient and fault tolerant, both in theory and in practice. Presents a distributed implementation of signSGD with majority vote as aggregation.