<p>There are two ways to train a BERT-based classification model:</p>
<ol>
<li><p><strong>Finetuning</strong>: Which is the practice of training your classifier along with your text encoder (BERT in this case, but it can be any other text encoder, e.g., RoBERTa, ALBERT...). In this setting, the encoder and the classifier are both trained at the same time.</p>
</li>
<li><p><strong>BERT as an embedding model</strong>: Here you freeze the weights of BERT, and you only train the classifier. At the end of such a setting, BERT would be exactly the same as before training.</p>
</li>
</ol>
<p>Research has shown that finetuning delivers slightly better results than when using BERT as embeddings. You can find the original research paper where these results are discussed <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">here</a>.</p>
<p>What you are suggesting though is a tradeoff between the two, and it is interesting. I never tried that out myself, but I suspect you will run into overfitting since you train your classifier twice on the same data. So I suppose you will have better results than freezing BERT, but your model will have harder time generalizing to unseen data than the finetuning method.</p>
<p>Yacine</p>
BERT finetuning : Is it right to train BERT Classification model at once?
<p>I have a question about training BERT classification(or pretrained model).</p>
<p>BERT classifier model usually constructed 2 models. BERT model and classifier.</p>
<p>Many BERT fine tuning example code is training BERT model and classifier layer at once.
But I think, classifier is training first and BERT weight should not updated. After classifier trained, training all model layers.</p>
<p>Example</p>
<pre class="lang-py prettyprint-override"><code>import torch
from transformers import BertForSequenceClassification
model = BertForSequenceClassification()
...

# training1
for name, param in model.named_parameters():
    if 'classifier' in name:
        param.requires_grad = True # only classifier update
    else:
        param.requires_grad = False # tied other layer

...
# And after training1, we can using BERT model that is trained only classfier.
model = BertForSequenceClassification()
model.load_state_dict(torch.load({model only trained classifier})
for name, param in model.named_parameters():
    param.requires_grad = True # training all 

# training BERT Classification model
</code></pre>
<p>Why BERT Classification model training at once?
Thank you.</p>