<p>100k words it too few to train such a large model as BERT or RoBERTa. The main claim of <a href="https://arxiv.org/pdf/1907.11692.pdf" rel="nofollow noreferrer">the RoBERTa paper</a> is that BERT is actually undertrained. Whereas BERT was trained on 16 GB of text data, RoBERTa used 160 GB of plain text.</p>

<p>For a small domain-specific data as you describe, you can try fine-tuning an existing model. In this case, I would go for RoBERTa because it seems to be better pre-trained, does not have the next-sentence-objective (which is a hassle to pre-process data for it) and it uses SentencePiece for tokenization, which allows loss-less detokenization.</p>
Pre-training BERT/RoBERTa language model using domain text, how long it gonna take estimately? which is faster?
<p>I want to pre-train BERT and RoBERTa MLM using domain corpus (sentiment-related text). How long it gonna take for using 50k~100k words. Since RoBERTa is not trained on predicting the next sentence objective, one training objective less than BERT and with larger mini-batches and learning rates, I assume RoBERTa will be much faster?</p>