<p>Thought I might point out that, arbitrarily making the batch size large (when you have large amounts of memory) can be bad sometimes in terms of the generalization of your model.</p>

<p>Reference: </p>

<p><a href="https://arxiv.org/pdf/1705.08741.pdf" rel="nofollow noreferrer">Train longer, generalize better</a></p>

<p><a href="https://arxiv.org/abs/1609.04836" rel="nofollow noreferrer">On Large-Batch Training for Deep Learning</a>. </p>
Training GPU on multiple minibatches in parallel with TensorFlow
<p>I am using TensorFlow 1.9, on an NVIDIA GPU with 3 GB of memory. The size of my minibatch is 100 MB. Therefore, I could potentially fit multiple minibatches on my GPU at the same time. So my question is about whether this is possible and whether it is standard practice.</p>

<p>For example, when I train my TensorFlow model, I run something like this on every epoch:</p>

<pre><code>loss_sum = 0
for batch_num in range(num_batches):
    batch_inputs = get_batch_inputs()
    batch_labels = get_batch_labels()
    batch_loss, _ = sess.run([loss_op, train_op], feed_dict={inputs: batch_inputs, labels: batch_labels})
    loss_sum += batch_loss
loss = batch_loss / num_batches
</code></pre>

<p>This iterates over my minibatches and performs one weight update per minibatch. But the size of <code>image_data</code> and <code>label_data</code> is only 100 MB, so the majority of the GPU is not being used.</p>

<p>One option would be to just increase the minibatch size so that the minibatch is closer to the 3 GB GPU capacity. However, I want to keep the same small minibatch size to help with optimisation.</p>

<p>So the other option might be to send multiple minibatches through the GPU in parallel, and perform one weight update per minibatch. Being able to send the minibatches in parallel would significantly reduce the training time.</p>

<p>Is this possible and recommended?</p>