title: How should the learning rate change as the batch size change?
body: <p>When I increase/decrease batch size of the mini-batch used in SGD, should I change learning rate? If so, then how?</p>
  
  <p>For reference, I was discussing with someone, and it was said that, when batch size is increased, the learning rate should be decreased by some extent. </p>
  
  <p>My understanding is when I increase batch size, computed average gradient will be less noisy and so I either keep same learning rate or increase it. </p>
  
  <p>Also, if I use an adaptive learning rate optimizer, like Adam or RMSProp, then I guess I can leave learning rate untouched.</p>
  
  <p>Please correct me if I am mistaken and give any insight on this.</p>