moinnadeem.com; MosaicML ×Databricks
Jacob Portes, Alexander Trott, Sam Havens, DANIEL KING, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle
BERT;Pretraining;Efficiency;FlashAttention;ALiBi
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining