<p>Remember that fine-tuning a pre-trained model like Bert usually requires a much smaller number of epochs than models trained from scratch. In fact <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">the authors of Bert recommend between 2 and 4 epochs</a>. Further training often translates to overfitting to your data and forgetting the pre-trained weights (see <em>catastrophic forgetting</em>).</p>
<p>In my experience, this affects small datasets especially as it's easy to overfit on them, even at the 2nd epoch. Besides, you haven't commented on your custom layers on top of Bert, but adding much complexity there might increase overfitting also -- note that the common architecture for text classification only adds a linear transformation.</p>
BERT + custom layer training performance going down with epochs
<p>I'm training a classification model with custom layers on top of BERT. During this, the training performance of this model is going down with increasing epochs ( after the first epoch ) .. I'm not sure what to fix here - is it the model or the data?</p>
<p>( for the data it's binary labels, and balanced in the number of data points for each label).</p>
<p>Any quick pointers on what the problem could be? Has anyone come across this before?</p>
<p>Edit: Turns out there was a mismatch in the transformers library and tf version I was using. Once I fixed that, the training performance was fine!</p>
<p>Thanks!</p>