<p><a href="https://arxiv.org/abs/1711.00489" rel="nofollow noreferrer">This paper</a> researches the relation of batch size and learning rate.
Instead of decaying the learning rate, they increase the batch size by the same factor.</p>

<blockquote>
  <p>It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times.</p>
</blockquote>

<p>In short, if you use a bigger batch size, you can use a larger learning rate to reduce training time.</p>
Using CPU vs GPU to train a model - Speed vs memory
<p>I am trying to train the model found at 
<a href="https://github.com/silicon-valley-data-science/RNN-Tutorial" rel="nofollow noreferrer">https://github.com/silicon-valley-data-science/RNN-Tutorial</a> With a dataset generated through <a href="https://github.com/jupiter126/Create_Speech_Dataset" rel="nofollow noreferrer">https://github.com/jupiter126/Create_Speech_Dataset</a> (around 340000 small wav audio samples with transcripts).<br /><br />
When I train with GPU the training goes relatively fast, however I can't set batch_train_size above 25 without reaching OOM.<br />
When I train with CPU, training is much slower, but I can easily set batch_train_size to 250 (probably up to 700 but didn't try yet).<br /><br /></p>

<p>I am confused on how the small batch size limit on GPU might affect training quality, or if raising the raising the number of epoch might cancel out that effect...<br />  In other words, 25 samples on 10000 epoch or 500 samples on 500 epoch? <br /> <br />
GPU is GTX 1060 with 6Gb ram, CPU is dual XEON 2630l v4 (2*10 hyperthreaded cores at 1.7Ghz) with 128Gb ram.<br /><br /></p>