Department of Electronic Engineering, Tsinghua University; Institute for Precision Medicine, Tsinghua University; SKLSDE, Institute of Artiﬁcial Intelligence, Beihang University; Department of Electronic Engineering, Tsinghua University
Jiaxi Wang, Ji Wu, Lei Huang
Batch Normalization;Transformer;Training Inference Discrepancy
Understanding the Failure of Batch Normalization for Transformers in NLP