title: focal loss in pytorch
body: <p>I have binary NLP classification problem and my data is very biased. Class 1 represents only 2% of data. For training I am oversampling from class 1 and for training my class distribution is 55%-45%. I have built a CNN. My last few layers and loss function as below</p>
  <pre><code>self.batch_norm2 = nn.BatchNorm1d(num_filters)
          
  self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)
          
  self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)
          
  self.fc3 = nn.Linear(fc2_neurons, 1)#changing on 6March - BCE with logits loss
  </code></pre>
  <p>Loss:</p>
  <pre><code>BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_examples[0]/class_examples[1])
      
  </code></pre>
  <p>In my evaluation function I am calling that loss as follows</p>
  <pre><code>loss=BCE_With_LogitsLoss(torch.squeeze(probs), labels.float())
  </code></pre>
  <p>I was suggested to use focal loss over <a href="https://stackoverflow.com/questions/71462326/pytorch-bcewithlogitsloss-calculating-pos-weight/71494404#71494404">here</a>.</p>
  <p>Please consider using <a href="https://stackoverflow.com/a/52161194/1714410">Focal loss</a>:
  Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár <a href="https://arxiv.org/abs/1708.02002" rel="nofollow noreferrer">Focal Loss for Dense Object Detection</a> (ICCV 2017).</p>
  <p>Is there any pytorch implementation of the same? I found few but now sure which are correct.
  one example - <a href="https://stackoverflow.com/questions/71300607/using-focal-loss-for-imbalanced-dataset-in-pytorch">Using Focal Loss for imbalanced dataset in PyTorch</a>
  another example - <a href="https://stackoverflow.com/questions/66178979/focal-loss-implementation">Focal loss implementation</a></p>
  <p>My questions:</p>
  <ol>
  <li>is there any specific implementation that I should use? i am using pytorch so would appreciate pytorch based solution.</li>
  <li>Should I use focal loss even though I am oversampling?</li>
  <li>How to modify my code to use the correct implementation</li>
  <li>do we need to use <code>pos_weight</code> along with it?</li>
  </ol>