<p>It's a <em>very</em> broad subject, but IMHO, you should try <a href="https://github.com/clcarwin/focal_loss_pytorch" rel="noreferrer">focal loss</a>: It was introduced by <a href="https://arxiv.org/pdf/1708.02002.pdf" rel="noreferrer">Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollar</a> to handle imbalance prediction in object detection. Since introduced it was also used in the context of segmentation.<br>
The idea of the focal loss is to reduce both loss and gradient for correct (or almost correct) prediction while emphasizing the gradient of errors.</p>

<p>As you can see in the graph:<br>
<a href="https://i.stack.imgur.com/aWvuD.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/aWvuD.jpg" alt="enter image description here"></a></p>

<p>Blue curve is the regular cross entropy loss: it has on the one hand non-negligible loss and gradient even for well classified examples, and on the other hand it has weaker gradient for the erroneously classified examples.<br>
In contrast, focal loss (all other curves) has smaller loss and weaker gradient for the well classified examples and stronger gradients for the  erroneously classified examples.</p>
Best loss function for multi-class classification when the dataset is imbalance?
<p>I'm currently using the Cross Entropy Loss function but with the imbalance data-set the performance is not great.</p>

<p>Is there better lost function?</p>