<p>Following <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="noreferrer">He <em>et. al</em></a> (as suggested in <a href="https://stackoverflow.com/users/2658050/lejlot">lejlot</a>'s comment), initializing the weights of the <em>l</em>-th layer to a zero-mean Gaussian distribution with standard deviation </p>

<p><img src="https://latex.codecogs.com/gif.latex?%5Csqrt%7B%5Cfrac%7B2%7D%7Bn_l%7D%7D"/></p>

<p>where <em>n<sub>l</sub></em> is the flattened length of the the input vector or</p>

<pre><code>stddev=np.sqrt(2 / np.prod(input_tensor.get_shape().as_list()[1:]))
</code></pre>

<p>results in weights that generally do not diverge.</p>
Why are my TensorFlow network weights and costs NaN when I use RELU activations?
<p>I can't get TensorFlow RELU activations (neither <code>tf.nn.relu</code> nor <code>tf.nn.relu6</code>) working without NaN values for activations and weights killing my training runs.</p>

<p>I believe I'm following all the right general advice. For example I initialize my weights with </p>

<pre><code>weights = tf.Variable(tf.truncated_normal(w_dims, stddev=0.1))
biases = tf.Variable(tf.constant(0.1 if neuron_fn in [tf.nn.relu, tf.nn.relu6] else 0.0, shape=b_dims))
</code></pre>

<p>and use a slow training rate, e.g.,</p>

<pre><code>tf.train.MomentumOptimizer(0.02, momentum=0.5).minimize(cross_entropy_loss)
</code></pre>

<p>But any network of appreciable depth results in <code>NaN</code> for cost and and at least some weights (at least in the summary histograms for them). In fact, the cost is often <code>NaN</code> right from the start (before training).</p>

<p>I seem to have these issues even when I use L2 (about 0.001) regularization, and dropout (about 50%).</p>

<p>Is there some parameter or setting that I should adjust to avoid these issues? I'm at a loss as to where to even begin looking, so any suggestions would be appreciated!</p>