<p>Adam is famous for working out of the box with its <em>default</em> paremeters, which, in almost all frameworks, include a learning rate of 0.001 (see the default values in <a href="https://keras.io/optimizers/" rel="nofollow noreferrer">Keras</a>, <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html" rel="nofollow noreferrer">PyTorch</a>, and <a href="https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer" rel="nofollow noreferrer">Tensorflow</a>), which is indeed the value suggested in the <a href="https://arxiv.org/abs/1412.6980" rel="nofollow noreferrer">Adam paper</a>.</p>

<p>So, I would suggest changing to </p>

<pre><code>optimizer = optim.Adam(model.parameters(), lr=0.001)
</code></pre>

<p>or simply</p>

<pre><code>optimizer = optim.Adam(model.parameters())
</code></pre>

<p>in order to leave <code>lr</code> in its default value (although I would say I am surprised, as MNIST is famous nowadays for working practically with whatever you may throw into it).</p>
Model learns with SGD but not Adam
<p>I was going through a basic PyTorch MNIST example <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="nofollow noreferrer">here</a> and noticed that when I changed the optimizer from SGD to Adam the model did not converge. Specifically, I changed line 106 from</p>

<pre><code>optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)
</code></pre>

<p>to</p>

<pre><code>optimizer = optim.Adam(model.parameters(), lr=args.lr)
</code></pre>

<p>I thought this would have no effect on the model. With SGD the loss quickly dropped to low values after about a quarter of an epoch. However with Adam, the loss did not drop at all even after 10 epochs. I'm curious to why this is happening; it seems to me these should have nearly identical performance.</p>

<p>I ran this on Win10/Py3.6/PyTorch1.01/CUDA9</p>

<p>And to save you a tiny bit of code digging, here are the hyperparams:</p>

<ul>
<li>lr=0.01</li>
<li>momentum=0.5</li>
<li>batch_size=64</li>
</ul>