<p>It is a known thing with high capacity models. They are suprisingly resistant to overfitting which contradicts to the classical statistical learning theory that says that without explicit regularization you going to overfit. For example, <a href="https://www.padl.ws/papers/Paper%2010.pdf" rel="nofollow noreferrer">this paper</a> says</p>

<blockquote>
  <p>most of deep neural networks with learned parameters often generalize
  very well empirically, even equipped with much more effective
  parameters than the number of training samples, i.e. high capacity...
  Thus, statistical learning theory cannot explain the generalization
  ability of deep learning models.</p>
</blockquote>

<p>Also, <a href="https://arxiv.org/pdf/1705.08741.pdf" rel="nofollow noreferrer">this</a> and <a href="https://arxiv.org/pdf/1710.10345.pdf" rel="nofollow noreferrer">this</a> papers are talking about it. You can keep on following the references in these papers to read more.</p>

<p>Personally, I have never seen high capacity model overfits even after training for 10s of thousands of epochs. If you want the example that does overfit: take Lenet 5 for Cifar10 with ReLU activations and without dropout and train it using SGD with learning rate <code>0.01</code>. The number of training parameters in this model is ~ 60000 thousand which is the same as the number of samples in Cifar10 (low capacity model). After at most 500-1000 epochs you are going to see a very clear overfitting with increasing loss and error over time.</p>
What does it mean if my network can never overfit no matter how much I train it or expand its capacity?
<p>I trained a model, got decent results, but then I got greedy and I wanted even more accuracy, so, I trained the model for longer, and longer and longer, but to no avail, nothing happens! according to theory, at some point, the validation accuracy must start to decrease after too much training (the loss start to INCREASE)! but this never seem to happen. So, I figured may be the NN is too simple to ever be able to overfit, so I increased its capacity and I ended up with millions of parameters, and I trained it for 10,000 epochs, still no overfitting happens. </p>

<p>The same question was asked <a href="https://stackoverflow.com/questions/54489549/why-does-my-neural-network-never-overfit">here</a>, but the answers there are anything but satisfying.</p>

<p>What does that mean?</p>