NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China; Collaborative Innovation Center for Language Ability, Jiangsu Normal University, China; ModelBest Inc., Beijing, China; NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China
Yuzhong Wang, Xu Han, Weilin Zhao, Guoyang Zeng, Zhiyuan Liu, Maosong Sun
ML System;Parallelism Learning;Memory Optimization;Data Parallelism;Model Parallelism;Parameter Parallelism;ZeRO;Rematerialization;Checkpointing;Tensor Offloading;Dynamic Programming
H3T: Efficient Integration of Memory Optimization and Parallelism for Large-scale Transformer Training