1HSE University; University of Washington; 2Yandex; 1HSE University, 2Yandex
Max Ryabinin, Tim Dettmers, Michael Diskin, Alexander Borzunov
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient