MIT EECS/LIDS; MIT LIDS; MIT CEE/LIDS; KAIST AI; KAIST ISysE/Math
Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Suvrit Sra, Ali Jadbabaie
Transformer;optimization;adam;clipping;heavy-tailed noise;directional smoothness
Linear attention is (maybe) all you need (to understand Transformer optimization)