<p>A drawback of SGD-based optimizers is that they rely upon scalar and uniform learning of gradients in all the directions (i.e., for all the parameters for which gradient is to be updated). In contrast, adaptive learning strategies such as Adam diagonally scale the gradient based upon estimates of the function’s curvature. Instead of maintaining a learning rate that is shared amongst multiple parameters, Adam uses a vector of learning rates, one for each parameter and adapts those as the training progresses. It is this non-uniform scaling of the gradients that results in the lag of generalization capabilites of Adam, and probably in your case, the massive decrease in accuracies.</p>

<p>As mentioned in <a href="https://arxiv.org/pdf/1712.07628.pdf" rel="nofollow noreferrer">Improving Generalization Performance by Switching from Adam to SGD</a>:</p>

<blockquote>
  <p>Despite superior training outcomes, adaptive optimization methods such
  as Adam, Adagrad or RMSprop have been found to generalize poorly
  compared to Stochastic gradient descent (SGD). These methods tend to
  perform well in the initial portion of training but are outperformed
  by SGD at later stages of training.</p>
</blockquote>

<p>In order to combine the best of both the optimizers, they introduce a switching technique from Adam to SGD by taking care of: (a) the switchover point, i.e. how long to train the model with Adam before switching to SGD. As a thumb rule, the paper points at switiching after at least 10 epochs. (b) the learning rate to be used for SGD after the switch: determined by the momentum parameter <code>beta_1</code> of Adam. </p>

<p>A good explanation can be found <a href="https://github.com/kweonwooj/papers/issues/76" rel="nofollow noreferrer">here</a>.</p>
Why does neural net optimizer change from SGD to Adam lead to massive decrease in accuracies?
<p>I'm training a covnet on ~10,000 images and have noticed that switching the optimizer from <code>opt = SGD()</code> to <code>opt = 'adam'</code> leads to massive reduction in accuracies, keeping all else params equal. With SGD(), I get to about 80% accuracy (with gradual increases after each epoch). With Adam, I'm stuck at 22.25% validation accuracies at every epoch. </p>

<p><strong>I want to understand what the likely cause for this is.</strong> </p>

<p>Parameters</p>

<pre><code>dropout_prob = 0.2
activation_function = 'relu'
loss_function = 'categorical_crossentropy'
batch_size = 32
epoch_count = 20
num_classes = 3
</code></pre>

<p>Model</p>

<pre><code> model = Sequential()
        model.add(Conv2D(filters=16, kernel_size=(3, 3), input_shape=inp_shape))
        model.add(Conv2D(filters=32, kernel_size=(3, 3)))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(rate=dropout_prob))
        model.add(Flatten())
        model.add(Dense(128, activation=activation_function))
        model.add(Dropout(rate=dropout_prob))
        model.add(Dense(64, activation=activation_function))
        model.add(Dropout(rate=dropout_prob))
        model.add(Dense(32, activation=activation_function))
        model.add(Dense(num_classes, activation='softmax'))
        model.summary()
        model.compile(loss=loss_function, optimizer=opt, metrics=['accuracy'])
        history = model.fit(x_train, y_train_cat, batch_size=batch_size, epochs=epoch_count,
                            verbose=verbose_level,
                            validation_data=(x_test, y_test_cat))
</code></pre>