title: Why distillation loss doesn't converge(decrease) compared to cross-entropy loss?
body: <p>I'm using cross entropy &amp;&amp; distillation loss for the continual learning(a.k.a. incremental learning) model.
  But, distillation loss doesn't converge while cross-entropy loss converges.</p>
  
  <p>Here is my code for the distillation loss and the training part.</p>
  
  <p>Code for distillation loss:</p>
  
  <pre><code>def loss_fn_distillation(outputs, soft_labels, temperature, current_step, total_step, total_label):
      current_label = (total_label / total_step) * (current_step + 1)
      previous_label = (total_label / total_step) * current_step
  
      soft_labels = V(soft_labels.data, requires_grad=False).cuda()
      soft_labels = torch.softmax(soft_labels / temperature, dim=1)
  
       outputs = F.log_softmax(outputs[:,:-int(current_label-previous_label)]/temperature, dim = 1)
  
       distill_loss = torch.sum(outputs * soft_labels, dim=1, keepdim=False)
       distill_loss = -torch.mean(distill_loss, dim=0, keepdim=False)
  
  
        return V(distill_loss, requires_grad=True).cuda()
  </code></pre>
  
  <p>Code for training part:</p>
  
  <pre><code>    outputs = net(inputs)
      ce_loss = criterion(outputs, targets)
  
      if(i&gt;0) :
  
          soft_label = previous_net(inputs)
  
          distill_loss = loss_fn_distillation(outputs=outputs, soft_labels=soft_label, temperature=2,
                                              current_step=i, total_step=step, total_label=number_label)
          print(ce_loss, distill_loss)
          loss = distill_loss + ce_loss
  
  
      else :
          loss = ce_loss
  
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
  </code></pre>
  
  <p>Result for cross entropy loss &amp;&amp; distillation loss for epoch:</p>
  
  <p><img src="https://i.stack.imgur.com/RsnLe.jpg" alt="enter image description here"></p>
  
  <p>I'd appreciate any feedback. Thank you.</p>