Key Laboratory of Machine Perception, MOE, School of EECS, Peking University; Princeton University; Institute for Artiﬁcial Intelligence, Peking University; Center for Data Science, Peking University; University of Science and Technology of China; Microsoft Research
Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, Tie-Yan Liu
Attention;Transformer;Relative Positional Encoding;Fast Fourier Transform;Language Pre-training;Language Modelling;Machine Translation;Image Classification
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding