Department of Computer Science, Stanford University; Department of Computer Science and Engineering, University at Buffalo, SUNY
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, Christopher Ré
Attention;GPUs;Hardware-efficient model;Long context;IO complexity
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness