<p>Some of the famous initializers for Convolutional Neural Networks:</p>

<p><strong>Glorot Normal</strong>: Also called Xavier. Normal distribution centered on 0 with stddev = sqrt(2 / (fan_in + fan_out)) where fan_in is the number of input units in the weight tensor and fan_out is the number of output units in the weight tensor.</p>

<p><a href="http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf</a></p>

<p><strong>Lecun Uniform</strong>: Uniform distribution within [-limit, limit] where limit is sqrt(3 / fan_in) where  fan_in is the number of input units in the weight tensor.</p>

<p><a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noreferrer">http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf</a></p>

<p><strong>He Normal</strong>:
Truncated normal distribution centered on 0 with stddev = sqrt(2 / fan_in) where  fan_in is the number of input units in the weight tensor.</p>

<p><a href="http://arxiv.org/abs/1502.01852" rel="nofollow noreferrer">http://arxiv.org/abs/1502.01852</a></p>

<p>Along with these initializers, one have to search for learning rate, momentum and other hyperparameters.</p>
Choosing initial values for variables and parameters for optimizers in tensorflow
<p>How do people typically choose initial values for their variables and parameters? Do we just tinker till it works?</p>

<p>I was following the Getting Started tutorial for tensorflow, and was able to train the linear model in it. However, I noticed that the starting values for the variables W, b were reasonably close to the ground truth.</p>

<p>When I change the data to make the ground truth values much further away, the gradient descent optimizer gives me NaN values for W, b.</p>

<p>However, in general, I don't think it would be reasonable to be able to guess the initial values of the variables in the model. Seems like I should be able to choose any arbitrary starting point and get to where I want.</p>

<p>I was thinking my choice in my parameters might be bad. However, I am not sure in what way to adjust this. The default was 0.01, I've tried values from 0.001 to 100.</p>

<p>Would there be a discussion of optimization parameter choices and initial values for model variables in a general machine learning book? Really I am just looking for resources.</p>

<p>Thanks!</p>