<p>I think that the best choice here is to use some probability ditribution pseudo-distance, the first choice that came to my mind is to use Kullback-Leiber Divergence, it is already implemented in pytorch and keras( see [kldivloss](<a href="https://pytorch.org/docs/stable/nn.html#kldivloss" rel="nofollow noreferrer">https://pytorch.org/docs/stable/nn.html#kldivloss</a> and <a href="https://keras.io/losses/#kullback_leibler_divergence" rel="nofollow noreferrer">keras</a>) Other famous ditances may include Jesnsen-Shanon divergence and Earth-Mover distance (This the same distance thatwas used in <a href="https://arxiv.org/abs/1701.07875" rel="nofollow noreferrer">WGAN</a></p>
What might be the best loss function when target is a gaussian label?
<p>I have a simple CNN with the inputs as </p>

<ul>
<li>Cropped grayscale patches of size MxN centered on the object of interest. The intensity of each patch is rescaled to [0, 1].</li>
<li>Target Gaussian label of the same size MXN with values ranging
in [5.0155e-173, 1]. This label is kept fixed throughout the training.</li>
</ul>

<p><a href="https://i.stack.imgur.com/fzyRs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fzyRs.png" alt="Target Gaussian label with (5.0155e-173, 1) as (min, max)"></a></p>

<p>The goal is to learn the target label and use the learned model to detect the object in a test image. I am using Adam optimizer with various loss functions such as <code>categorical_crossentropy</code>, <code>mean_squared_error</code>, and <code>mean_absolute_error</code> but training halts soon probably due to the low values returned by all these loss functions (vanishing gradients?). Increasing the batch size from 1 to 16~32 sometimes helps in completing the iteration but gives undesired outcomes at test time.</p>

<p>Is it because the loss function is too sensitive to the lower values in the target and even treats them as outliers hence steering the whole learning process in the wrong direction?</p>

<p>I'll be grateful for your help in fixing the loss function in such a scenario.</p>