- Given the existing literature on the implicit bias of optimization methods, the primary concern is the significance of the results presented. For instance, the classic result by [Z. Ji and M. Telgarsky] demonstrates a convergence rate $\log\log n/\log n$ of GD to the L2-margin solution, which is faster than the rate shown in this submission. Moreover, [C. Zhang, D. Zou, and Y. Cao] have shown much faster rates for Adam converging to the L-infinity margin solution. This submission also lacks citations to these papers and other relevant works:
[Z. Ji and M. Telgarsky] The implicit bias of gradient descent on nonseparable data, COLT 2019.
[C. Zhang, D. Zou, and Y. Cao] The Implicit Bias of Adam on Separable Data. 2024.
[S. Xie and Z. Li] Implicit Bias of AdamW: l_\infty-Norm Constrained Optimization. ICML 2024
[M. Nacson, N. Srebro, and D. Soudry] Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. AISTATS 2019.
- Since AdaGrad-Norm has the same implicit bias as GD, the advantages of using AdaGrad-Norm over GD are unclear.
- The bounded noise assumption, while common, is somewhat restrictive in stochastic optimization literature. There have been several efforts to extend these noise conditions:
[A. Khaled and P. Richt´arik]. Better theory for sgd in the nonconvex world. TMLR 2023.
[R. Gower, O. Sebbouh, and N. Loizou] Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. AISTATS 2021.
- The bounded noise assumption, while common, is somewhat restrictive in stochastic optimization literature. There have been several efforts to extend these noise conditions: [A. Khaled and P. Richt´arik]. Better theory for sgd in the nonconvex world. TMLR 2023. [R. Gower, O. Sebbouh, and N. Loizou] Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. AISTATS 2021.