University of British Columbia, Canada; University of British Columbia, Canada CIFAR AI Chair (Amii)
Frederik Kunstner, Jacques Chen, Jonathan Lavington, Mark Schmidt
optimization for deep learning;adaptive methods;adam;rmsprop;sgd;sign descent;noise;stochasticity;full batch
Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be