<p>BERT can be viewed as a language encoder, which is trained on a humongous amount of data to learn the language well. As we know, the original BERT model was trained on the entire English Wikipedia and Book corpus, which sums to <strong>3,300M</strong> words. BERT-base has 109M model parameters. So, if you think you have large enough data to train BERT, then the answer to your question is yes. </p>

<p>However, when you said "still achieve a good result", I assume you are comparing against the original BERT model. In that case, the answer lies in the size of the training data.</p>

<p>I am wondering why do you prefer to train BERT from scratch instead of fine-tuning it? Is it because you are afraid of the domain adaptation issue? If not, pre-trained BERT is perhaps a better starting point.</p>

<p>Please note, if you want to train BERT from scratch, you may consider a <strong>smaller</strong> architecture. You may find the following papers useful.</p>

<ul>
<li><a href="https://arxiv.org/abs/1908.08962" rel="noreferrer">Well-Read Students Learn Better: On the Importance of Pre-training Compact Models</a></li>
<li><a href="https://arxiv.org/abs/1909.11942" rel="noreferrer">ALBERT: A Lite BERT for Self-supervised Learning of Language Representations</a></li>
</ul>
Can you train a BERT model from scratch with task specific architecture?
<p>BERT pre-training of the base-model is done by a language modeling approach, where we mask certain percent of tokens in a sentence, and we make the model learn those missing mask. Then, I think in order to do downstream tasks, we add a newly initialized layer and we fine-tune the model.</p>

<p>However, suppose we have a gigantic dataset for sentence classification. Theoretically, can we initialize the BERT base architecture from scratch, train both the additional downstream task specific layer + the base model weights form scratch with this sentence classification dataset only, and still achieve a good result?</p>

<p>Thanks.</p>