<p>Maybe the poor performance is due to gradients being applied to the BERT backbone. Validate it like so:</p>
<pre><code>print([p.requires_grad for p in bert_distil.distilbert.parameters()])
</code></pre>
<p>As an alternative solution, try freezing the weights of your trained model:</p>
<pre><code>for param in bert_distil.distilbert.parameters():
    param.requires_grad = False
</code></pre>
<p>As you are trying to optimize the weights of a trained model during fine-tuning on your data, you face issues described, among other sources, in the ULMIfit (<a href="https://arxiv.org/abs/1801.06146" rel="nofollow noreferrer">https://arxiv.org/abs/1801.06146</a>) paper</p>
Fine-Tuning DistilBertForSequenceClassification: Is not learning, why is loss not changing? Weights not updated?
<p>I am relatively new to PyTorch and Huggingface-transformers and experimented with DistillBertForSequenceClassification on this <a href="https://www.kaggle.com/c/nlp-getting-started" rel="nofollow noreferrer">Kaggle-Dataset</a>.</p>
<pre><code>from transformers import DistilBertForSequenceClassification
import torch.optim as optim
import torch.nn as nn
from transformers import get_linear_schedule_with_warmup

n_epochs = 5 # or whatever
batch_size = 32 # or whatever

bert_distil = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
#bert_distil.classifier = nn.Sequential(nn.Linear(in_features=768, out_features=1), nn.Sigmoid())
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(bert_distil.parameters(), lr=0.1)

X_train = []
Y_train = []

for row in train_df.iterrows():
    seq = tokenizer.encode(preprocess_text(row[1]['text']),  add_special_tokens=True, pad_to_max_length=True)
    X_train.append(torch.tensor(seq).unsqueeze(0))
    Y_train.append(torch.tensor([row[1]['target']]).unsqueeze(0))
X_train = torch.cat(X_train)
Y_train = torch.cat(Y_train)

running_loss = 0.0
bert_distil.cuda()
bert_distil.train(True)
for epoch in range(n_epochs):
    permutation = torch.randperm(len(X_train))
    j = 0
    for i in range(0,len(X_train), batch_size):
        optimizer.zero_grad()
        indices = permutation[i:i+batch_size]
        batch_x, batch_y = X_train[indices], Y_train[indices]
        batch_x.cuda()
        batch_y.cuda()
        outputs = bert_distil.forward(batch_x.cuda())
        loss = criterion(outputs[0],batch_y.squeeze().cuda())
        loss.requires_grad = True
   
        loss.backward()
        optimizer.step()
   
        running_loss += loss.item()  
        j+=1
        if j == 20:   
            #print(outputs[0])
            print('[%d, %5d] running loss: %.3f loss: %.3f ' %
              (epoch + 1, i*1, running_loss / 20, loss.item()))
            running_loss = 0.0
            j = 0
</code></pre>
<blockquote>
<p>[1,   608] running loss: 0.689 loss: 0.687
[1,  1248] running loss: 0.693 loss: 0.694
[1,  1888] running loss: 0.693 loss: 0.683
[1,  2528] running loss: 0.689 loss: 0.701
[1,  3168] running loss: 0.690 loss: 0.684
[1,  3808] running loss: 0.689 loss: 0.688
[1,  4448] running loss: 0.689 loss: 0.692  etc...</p>
</blockquote>
<p>Regardless on what I tried, loss did never decrease, or even increase, nor did the prediction get better. It seems to me that I forgot something so that weights are actually not updated. Someone has an idea?
O</p>
<p><strong>what I tried</strong></p>
<ul>
<li>Different loss functions
<ul>
<li>BCE</li>
<li>CrossEntropy</li>
<li>even MSE-loss</li>
</ul>
</li>
<li>One-Hot Encoding vs A single neuron output</li>
<li>Different learning rates, and optimizers</li>
<li>I even changed all the targets to only one single label, but even then, the network did'nt converge.</li>
</ul>