Feng Chen, Daniel Kunin, Atsushi Yamamura, Surya Ganguli
Implicit Bias;SGD Dynamics;Implicit regularization;Learning rate schedule;Stochastic Gradient Descent;Invariant set;Attractive saddle points;Stochastic collapse;Permutation invariance;Simplicity bias;Teacher-student
Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks