<p>It seems like you are facing a severe &quot;class imbalance&quot; problem.</p>
<ol>
<li><p>Have a look at <a href="https://stackoverflow.com/a/52161194/1714410">focal loss</a>. This loss is designed for binary classification with severe class imbalance.</p>
</li>
<li><p>Consider &quot;hard negative mining&quot;: that is, propagate gradients only for part of the training examples - the &quot;hard&quot; ones.<br />
see, e.g.:<br />
<em>Abhinav Shrivastava, Abhinav Gupta and Ross Girshick</em> <strong><a href="https://arxiv.org/abs/1604.03540" rel="nofollow noreferrer">Training Region-based Object Detectors with Online Hard Example Mining</a></strong> (CVPR 2016).</p>
</li>
</ol>
Training Sparse Autoencoders
<p>My dataset consists of vectors that are massive.  The data points are all mostly zeros with ~3% of the features being 1.  Essentially my data is super sparse and I am attempting to train an autoencoder however my model is learning just to recreate vectors of all zeros.</p>
<p>Are there any techniques to prevent this?  I have tried replacing mean squared error with dice loss but it completely stopped learning.  My other thoughts would be to use a loss function that favors guessing 1s correctly rather than zeros.  I have also tried using a sigmoid and linear last activation with no clear winner.  Any ideas would be awesome.</p>