Arvind Mahankali, Tatsunori Hashimoto, Tengyu Ma
Linear Self-Attention;In-context learning;Gradient Descent;Theoretical Understanding
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention