Meta AI
LILI YU, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis
byte level language model;model architecture;efficient pretraining
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers