<p>Okay. I know that you're more concerned about throughput from Keras and your hardware, but there are a few things I'd like to mention here:</p>

<ol>
<li><strong>smaller batch sizes gave me better results</strong></li>
</ol>

<p>Given you case, where you have not so huge data, assuming you're running the training for fixed number of epochs (say 5), training with lesser batch size is naturally expected to give you a slightly better result as it would mean a higher number of back-prop steps overall compared to that of a higher batch-size. If you're training for a fixed number of training steps instead, I don't know why this is happening. </p>

<ol start="2">
<li><strong>loss values will occasionally flip to NaN with batch sizes of 4</strong> </li>
</ol>

<p>Again, I'm assuming you're using batch-normalization here, with CNNs. While using BN, it's never actually recommended to use a smaller batch-size like 2 or 4 (or even 8). And probably, one of the reasons why you can be facing NaN with smaller batch-size is if you have low-variance in the current batch and if you take the epsilon value too small, you might have very small values that can lead to numerical instability going forward. But more generally, this might be a case of gradient instability like you mentioned. Consider using gradient clipping to see if it helps.</p>

<ol start="3">
<li><strong>GPU Workflow</strong> </li>
</ol>

<p>Here, I assume that you have only 1 GPU. And unfortunately, you can't parallelise using single-GPU. To clarify, you shouldn't be concerned about the size of your data for GPU RAM. In most of the single-GPU cases, the current batch stays on the CPU and GPU would only take up the operations. Rather, you should be concerned about the size of parameters that GPU would be computing. Since for 1-layer experiment and 3-layers experiment your operations differ a lot, I don't think it's possible as you can't place multiple ops on same device simultaneously. The best case for you here would be to use a larger batch-size (not too large - as this would reduce the number of back-prop steps in case of training for fixed-epochs), so that you'd cover more data in a single-go.</p>

<p>Just a tip for hyper-paramter tuning, you can consider using <a href="https://arxiv.org/abs/1507.06228" rel="nofollow noreferrer">Highway-CNNs</a>. These are inspired from gating mechanism of LSTMs where you specify a large number of hidden layers and the network figures out itself on how to control the information flow among the layers. So in short, this would practically eliminate your efforts of tuning the depth of network, and allowing you tune other hyper-params like learning rate or filter-sizes etc. </p>

<p>I hope at least some of this is relevant and helpful to you ;)</p>
Improving cross-validation throughput in TensorFlow, Keras
<p>I am working on CNN models which are intended to predict a protein's structure from its amino acid sequence.  I am implementing my CNN's in Keras.  The Keras API is the one that comes bundled with TensorFlow 1.4.0, so obviously TensorFlow is my backend.  I have installed the GPU version of TensorFlow, and I have verified that the GPU is being used.  My GPU is somewhat older, an NVidia GTX 760.</p>

<p>When I perform 3X cross-validation to help select architectures and hyperparameters, I have 50K examples in my training folds and 25K samples in my validation folds.  These are decently large data sets, however they're small in comparison to the RAM available in my computer (16 GB) or on my GPU (2 GB).  Fully unpacked and expressed as float32 values, with redundancy introduced because of sliding windows, all the folds taken together, input plus target values, occupies 316 MB.  I have pre-calculated my folds, and saved files of each fold to disk.  When I experiment with architectures and hyperparameters, the same folds are being used in every trial.</p>

<p>I started with networks containing a single hidden layer to see what I could achieve, and then switched to two hidden layers.  I used a fixed batch size of 64 for all of my early experiments.  Training proceeded quickly enough that I didn't concern myself with speed.  Performing a 3X cross-validation for a given architecture typically took about 12 minutes.</p>

<p>But in the last experiment that I did with two-layer networks, I decided to start investigating the effect of batch size.  I learned that smaller batch sizes gave me better results, up to a point.  Batch sizes of 8 were the smallest ones that I could count on not to crash.  My loss values will occasionally flip to NaN with batch sizes of 4, and they will frequently flip to NaN with batch sizes of 1 or 2.  After that occurs, the network becomes untrainable.  I am aware of the possibility of gradient instability.  I think I was getting some.</p>

<p>So why not just use batch sizes of 8 and keep going?  The problem is speed.  Using two hidden layers, batches of eight took me approximately 35 minutes to cross-validate.  Batches of 64, as I mentioned above, took one third that much time.  My first experiments with three hidden layers have taken 45 to 65 minutes per trial.  And I want to investigate potentially hundreds of architectures and hyperparameters, using still deeper networks.  With small batches, I can see that the batch-by-batch progress bar in Keras progresses more slowly.  I can see much longer pauses when an epoch ends.</p>

<p>Yes, I can upgrade my GPU to a 10 series.  I think that will only double my throughput at most?  Yes, I can rent GPU time in the cloud.  Eventually I might do that.  But if my software is inefficient, I definitely don't want to set it loose in the cloud to burn my money.</p>

<p>It is my understanding (please correct me if I am wrong) that when the GPU is used in a normal TF / Keras work flow, each individual batch is sent separately from the GPU to the CPU.  If I am training 50 networks in a 3X cross-validation scheme, this would mean that I'm sending the same data to my GPU 150 times.  As I mentioned earlier, all my data occupies at most 316 MB, about 15% of the RAM available on the GPU.  Can I devise a workflow which sends this 316 MB to the GPU <strong>once</strong>, and if so, will that have a useful impact on my throughput?  Intuitively, it feels like it should.</p>

<p>Are there other bottlenecks I should be thinking about?  Is there a way to profile TF or Keras operations?</p>

<p>Thanks for any advice you may have!</p>