<p>I think you must be using the <code>MSECriterion()</code>? It is the standard <em>l2</em> (minimum square error) loss. While the CNN tries to predict results, there are multiple modes through which the result can be correct. And what <em>l2</em> loss does is that it converges to an average of all these modes as that is the most feasible way it can intuitively approach to attain less-penalized results.</p>

<blockquote>
  <p>The MSE-based solution
  appears overly smooth due to the pixel-wise average of
  possible solutions in the pixel space</p>
</blockquote>

<p>To pick the optimum mode of answer, you can look into <code>adversarial loss</code> <a href="https://www.google.co.in/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwiE56G2oqzTAhVLrY8KHeB8DuIQFggiMAA&amp;url=https%3A%2F%2Farxiv.org%2Fabs%2F1406.2661&amp;usg=AFQjCNEYi3th_8xRH0BIjxxmc-AM4lKdzA&amp;sig2=XzUiulxFYpqgyRHFKQ3N1A" rel="nofollow noreferrer">LINK</a>. This loss picks the optimum mode based on what it thinks is realistic in terms of the data it has seen.</p>

<p>For further clarification, look at figure 3 in this paper: <a href="https://arxiv.org/pdf/1609.04802.pdf" rel="nofollow noreferrer">SRGAN</a></p>
How to prevent converge to mean solution for regression problems in CNN?
<p>I am training a CNN for predicting joints on hands. The problem is that my net always converges to the mean value of the training set, and I can only get identical results for different test images. Do you know how to prevent this?</p>