Shanghai AI Laboratory; Australian National University; SenseTime Research, Shanghai AI Laboratory; SenseTime Research
Qin Zhen, Weixuan Sun, Hui Deng, DONGXU LI, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, Yiran Zhong
Linear Transformer;softmax attention
cosFormer: Rethinking Softmax In Attention