<p>in my experience, NaNs, when training a network usually happen because of two problems:</p>

<ul>
<li>first, mathematical error, e.g. log of negative value. It could happen when you are using log() in your loss function.</li>
<li>Second, there is a value that becomes too big so python can't handle.</li>
</ul>

<p>In your case, from your good observation, I think it's a second case. Your loss value may become too big to handled by python. Try to initialize smaller weight when you try to expand your network. Or just use different approach to initialize weight like explained by <a href="http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf" rel="nofollow">Glorot (2010)</a> or <a href="http://arxiv.org/abs/1502.01852" rel="nofollow">He (2015)</a>. Hope it helps.</p>
Why does Theano throw NaNs when I use dropouts?
<p>I am training a simple feed-forward model with 3 or 4 hidden layers and dropouts between each (hidden layer + non linearity) combination.
Sometimes after a few epochs (about 10-11) the model starts outputting Infs and NaNs as the error of the NLL and the accuracy falls to 0.0%.  This problem does not happen when I do not use dropouts. Is this a known issue with dropouts in Theano? The way I implement dropouts is:</p>

<pre><code>def drop(self, input):
    mask = self.theano_rng.binomial(n=1, p=self.p, size=input.shape, dtype=theano.config.floatX)
    return input * mask
</code></pre>

<p>where input is the feature-vector on which we want to apply dropouts.
I have also observed that the occurance of NaNs happens earlier if the dropout probability (self.p) is higher. p = 0.5 would cause NaNs to occur around epoch 1 or 2 but p = 0.7 would cause NaNs to occur around epoch 10 or 11.
Also the occurrence of NaNs happens only when hidden layer sizes are large. For example (800,700,700) gives NaNs whereas (500,500,500) does not.</p>