Microsoft DeepSpeed; Snowflake; University of Houston; OpenAI; University of Nevada, Reno
Guanhua Wang, Heyang Qin, Sam Jacobs, Xiaoxia (Shirley) Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
low-precision LLM pretraining;2 bits;auto compression;low memory pretraining
ZeRO++: Extremely Efficient Collective Communication for Large Model Training