<p>As Ivan already noted you have a class imbalance problem. This can be resolved via:</p>
<ol>
<li><p><strong>Online hard negative mining:</strong> at each iteration after computing the loss, you can sort all elements in the batch belonging to &quot;no DR&quot; class and keep only the worst <code>k</code>. Then you estimate the gradient only using these worse k and <em>discard</em>  all the rest.<br />
see, e.g.:<br />
<em>Abhinav Shrivastava, Abhinav Gupta and Ross Girshick</em> <a href="https://arxiv.org/abs/1604.03540" rel="nofollow noreferrer"><strong>Training Region-based Object Detectors with Online Hard Example Mining</strong></a> (CVPR 2016)</p>
</li>
<li><p><a href="https://stackoverflow.com/a/52161194/1714410"><strong>Focal loss:</strong></a> a modification for the &quot;vanilla&quot; cross entropy loss can be used to tackle class imbalance.</p>
</li>
</ol>
<hr />
<p>Related posts <a href="https://stackoverflow.com/a/58213245/1714410">this</a> and <a href="https://stackoverflow.com/a/64365532/1714410">this</a>.</p>
Why do sometimes CNN models predict just one class out of all others?
<p>I am relatively new to the deep learning landscape, so please don't be as mean as Reddit! It seems like a general question so I won't be giving my code here as it doesn't seem necessary (if it is, here's the link to <a href="https://colab.research.google.com/drive/1x6DOqow1dnZvy29_UZ3nHAKZLaNKr9hM?usp=sharing" rel="nofollow noreferrer">colab</a>)</p>
<p>A bit about the data: You can find the original data <a href="https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-resized" rel="nofollow noreferrer">here</a>. It is a downsized version of the original dataset of 82 GB.</p>
<p>Once I trained my CNN on this, it predicts 'No Diabetic Retinopathy' (No DR) every single time, leading to an accuracy of 73%. Is the reason for this is just the vast amount of No DR images or something else? I have no idea! The 5 classes I have for prediction are <code>[&quot;Mild&quot;, &quot;Moderate&quot;, &quot;No DR&quot;, &quot;Proliferative DR&quot;, &quot;Severe&quot;]</code>.</p>
<p>It's probably just bad code, was hoping you guys could help</p>