The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn’t. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings. Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails.
Analyzing the popular Adam optimizer The paper trys to improve Adam based on variance adaption with momentum by proposing two algorithms This paper analyzes the scale-invariance and the particular shape of the learning rate used in Adam, arguing that Adam's update is a combination of a sign-update and a variance-based learning rate. The paper splits ADAM algorithm into two components: stochastic direction in sign of gradient and adaptive stepwise with relative variance, and two algorithms are proposed to test each of them.