<p>You can further pretrain a BERT model with your own data with run_mlm.py at: <a href="https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling" rel="nofollow noreferrer">https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling</a>.</p>
<p>Also look at this: <a href="https://github.com/allenai/dont-stop-pretraining" rel="nofollow noreferrer">https://github.com/allenai/dont-stop-pretraining</a> and the paper: <a href="https://arxiv.org/pdf/2004.10964.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2004.10964.pdf</a> for related ideas and terminology: domain-adaptive pertaining and task-adaptive
pretraining.</p>
How to further pretrain a bert model using our custom data and increase the vocab size?
<p>I am trying to further pretrain the bert-base model using the custom data. The steps I'm following are as follows:</p>
<ol>
<li><p>Generate list of words from the custom data and add these words to the existing bert-base vocab file. The vocab size has been increased from <code>35022</code> to <code>35880</code>.</p>
</li>
<li><p>I created the input data using <strong>create_pretraining_data.py</strong> from <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">the bert official github page</a>.</p>
</li>
<li><p>Doing the pretrain using <strong>run_pretraining.py</strong> but facing the mismatch error:</p>
</li>
</ol>
<blockquote>
<p>ValueError: Shape of variable bert/embeddings/word_embeddings:0
((35880, 128)) doesn't match with shape of tensor
bert/embeddings/word_embeddings ([30522, 128]) from checkpoint reader.</p>
</blockquote>
<p><strong>Note:</strong> I changed the <code>bert_config</code> file with lastest <code>vocab_size</code> as <code>35880</code>.</p>
<p>Please help me to understand the error and what changes should be made, so that I can pretrain with the custom vocab file.</p>