Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev; Department of Computer and Information Science, University of Pennsylvania
language models;efficient pre-training;masking policy
Bootstrapping Small \& High Performance Language Models with Unmasking-Removal Training Policy