§ INTRODUCTION

§ PRELIMINARIES AND NOTATIONS

§ DYNAMICS OF GRADIENT DESCENT ON DEGREE-4 MODEL

§ DIFFERENCES IN DEGREE-2 AND HIGHER DEGREE MODELS

§ GLOBAL TRAJECTORY AND CHAOS

§ CONNECTION TO REAL-WORLD MODELS

§ DISCUSSION AND CONCLUSION

§ ACKNOWLEDGMENTS

This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER), CCF-1934964 (Tripods) and a Sloan Research Fellowship.

[Ahn et al., 2022]
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra.
Understanding the unstable convergence of gradient descent.
arXiv preprint arXiv:2204.01050, 2022.

[Arora et al., 2018]
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu.
Theoretical analysis of auto rate-tuning by batch normalization.
arXiv preprint arXiv:1812.03981, 2018.

[Arora et al., 2022]
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi.
Understanding gradient descent on edge of stability in deep learning.
arXiv preprint arXiv:2205.09745, 2022.

[Chen & Bruna, 2022]
Lei Chen and Joan Bruna.
On gradient descent convergence beyond the edge of stability.
arXiv preprint arXiv:2206.04172, 2022.

[Cohen et al., 2021]
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar.
Gradient descent on neural networks typically occurs at the edge of
arXiv preprint arXiv:2103.00065, 2021.

[Damian et al., 2021]
Alex Damian, Tengyu Ma, and Jason D Lee.
Label noise sgd provably prefers flat global minimizers.
Advances in Neural Information Processing Systems,
34:0 27449–27461, 2021.

[Glorot & Bengio, 2010]
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural
In Proceedings of the thirteenth international conference on
artificial intelligence and statistics, pp.  249–256. JMLR Workshop and
Conference Proceedings, 2010.

[Jastrzebski et al., 2020]
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek
Tabor, Kyunghyun Cho, and Krzysztof Geras.
The break-even point on optimization trajectories of deep neural
arXiv preprint arXiv:2002.09572, 2020.

[Krizhevsky et al., 2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[Lewkowycz et al., 2020]
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy
The large learning rate phase of deep learning: the catapult
arXiv preprint arXiv:2003.02218, 2020.

[Li et al., 2021]
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora.
What happens after sgd reaches zero loss?–a mathematical framework.
arXiv preprint arXiv:2110.06914, 2021.

[Li et al., 2022]
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv
Robust training of neural networks using scale invariant
In International Conference on Machine Learning, pp.  12656–12684. PMLR, 2022a.

[Li et al., 2022]
Zhouzi Li, Zixuan Wang, and Jian Li.
Analyzing sharpness along gd trajectory: Progressive sharpening and
edge of stability.
arXiv preprint arXiv:2207.12678, 2022b.

[Lyu et al., 2022]
Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora.
Understanding the generalization benefit of normalization layers:
Sharpness reduction.
arXiv preprint arXiv:2206.07085, 2022.

[Ma et al., 2022]
Chao Ma, Lei Wu, and Lexing Ying.
The multiscale structure of neural network loss functions: The effect
on optimization and origin.
arXiv preprint arXiv:2204.11326, 2022.

[Papyan, 2018]
Vardan Papyan.
The full spectrum of deepnet hessians at scale: Dynamics with sgd
training and sample size.
arXiv preprint arXiv:1811.07062, 2018.

[Ruiz-Garcia et al., 2021]
Miguel Ruiz-Garcia, Ge Zhang, Samuel S Schoenholz, and Andrea J Liu.
Tilting the playing field: Dynamical loss functions for machine
In International Conference on Machine Learning, pp.  9157–9167. PMLR, 2021.

[Sagun et al., 2016]
Levent Sagun, Leon Bottou, and Yann LeCun.
Eigenvalues of the hessian in deep learning: Singularity and beyond.
arXiv preprint arXiv:1611.07476, 2016.

[Vershynin, 2018]
Roman Vershynin.
High-dimensional probability: An introduction with applications
in data science, volume 47.
Cambridge university press, 2018.

[Wang et al., 2022]
Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao.
Large learning rate tames homogeneity: Convergence and balancing
International Conference on Learning Representations, 2022.

[Wu et al., 2020]
Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, and Rong Ge.
Dissecting hessian: Understanding common structure of hessian in
neural networks.
arXiv preprint arXiv:2010.04261, 2020.

[Xing et al., 2018]
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio.
A walk with sgd.
arXiv preprint arXiv:1802.08770, 2018.

[Yao et al., 2020]
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.
Pyhessian: Neural networks through the lens of the hessian.
In 2020 IEEE international conference on big data (Big data),
pp.  581–590. IEEE, 2020.

§ ADDITIONAL EXPERIMENTS