1)The main assumption is borrowed from other works but is actually rarely used in the optimization field. Moreover, the benefits of this assumption is not well investigated. For example, a) why it is more reasonable than the previous one? B) why it can add gradient norm L_1 \nabla f(w_1) in Eqn (3) or why we do not add other term? It should be mentioned that a milder condition does not mean it is better, since it may not reflect the truth. For me, problem B) is especially important in this work, since the authors do not well explain and investigate it.
2)Results in Theorem 1 show that Adam actually does not converge, since this is a constant term O(D_0^{0.5}\delta) in Eqn. (5). This is not intuitive, the authors claim it is because the learning rate may not diminish. But many previous works, e.g. [ref 1], can prove Adam-type algorithms can converge even using a constant learning rate. Of course, they use the standard smooth condition. But (L0,L1)-smoothness condition should not cause this kind of convergence, since for nonconvex problem, in most cases, we only need the learning rate to be small but does not care whether it diminishes to zero.
[ref 1] Dongruo Zhou, Jinghui Chen, et al. On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
3)It is not clear what are the challenges when the authors analyze Adam under the (L0,L1)-smoothness condition. It seems one can directly apply standard analysis on the (L0,L1)-smoothness condition. So it is better to explain the challenges, especially the difference between this one and Zhang et al.
4)Under the same assumption, the authors use examples to show the advantage of Adam over GD and SGD. This is good. But one issue is that is the example reasonable or does it share similar properties with practical problems, especially for networks. This is important since both SGD and ADAM are widely used in the deep learning field.
5)In the work, when comparing SGD and ADAM, the authors explain the advantage of adam comes from the cases when the local smoothness varies drastically across the domain. It is not very clear for me why Adam could better handle this case. Maybe one intuitive example could help.
6)The most important problem is that this work does not provide new insights, since it is well known that the second order moment could help the convergence of Adam. This work does not provide any insights beyond this point and also does not give any practical solution to further improve.
3)It is not clear what are the challenges when the authors analyze Adam under the (L0,L1)-smoothness condition. It seems one can directly apply standard analysis on the (L0,L1)-smoothness condition. So it is better to explain the challenges, especially the difference between this one and Zhang et al.