<p>Theory suggests that when multiplying the batch size by k, one should multiply the learning rate by sqrt(k) to keep the variance in the gradient expectation constant. See page 5 at <em>A. Krizhevsky.   One weird trick for parallelizing convolutional neural networks</em>: <a href="https://arxiv.org/abs/1404.5997" rel="noreferrer">https://arxiv.org/abs/1404.5997</a></p>

<p>However, recent experiments with large mini-batches suggest for a simpler linear scaling rule, i.e multiply your learning rate by k when using mini-batch size of kN.
See <em>P.Goyal et al.: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour</em> <a href="https://arxiv.org/abs/1706.02677" rel="noreferrer">https://arxiv.org/abs/1706.02677</a></p>

<p>I would say that with using Adam, Adagrad and other adaptive optimizers, learning rate may remain the same if batch size does not change substantially.</p>
How should the learning rate change as the batch size change?
<p>When I increase/decrease batch size of the mini-batch used in SGD, should I change learning rate? If so, then how?</p>

<p>For reference, I was discussing with someone, and it was said that, when batch size is increased, the learning rate should be decreased by some extent. </p>

<p>My understanding is when I increase batch size, computed average gradient will be less noisy and so I either keep same learning rate or increase it. </p>

<p>Also, if I use an adaptive learning rate optimizer, like Adam or RMSProp, then I guess I can leave learning rate untouched.</p>

<p>Please correct me if I am mistaken and give any insight on this.</p>