IDEAS NCBR, Institute of Mathematics, Polish Academy of Sciences, deepsense.ai; xAI, Work done while at Google Research; University of Warsaw, Google DeepMind; IDEAS NCBR, University of Warsaw
Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, Piotr Miłoś
Transformers;Language Models;Natural Language Processing
Focused Transformer: Contrastive Training for Context Scaling