School of Computer Science, Peking University, Beijing, China 100871; BDBC, Beihang University, Beijing, China 100191
Haoyi Zhou, Siyang Xiao, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li
Neural Network;Transformer;Self-attention
Jump Self-attention: Capturing High-order Statistics in Transformers