The Hebrew University of Jerusalem
Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua
Language Modeling;Pretraining;Self-attention;Transformers;Expressivity;Separation Rank;Sentence Embeddings
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design