<ul>
<li><p><strong>All you need is a good init (2016)</strong> : This paper proposes a simple method for weight initialization for deep net learning (<a href="http://arxiv.org/abs/1511.06422" rel="noreferrer">http://arxiv.org/abs/1511.06422</a>)</p></li>
<li><p>Watch this 6 mins video by andrew ng (Machine Learning, Coursera -> Week 5-> Random Initialization) explains danger of setting all initial weights to zero in Backpropagation (<a href="https://www.coursera.org/learn/machine-learning/lecture/ND5G5/random-initialization" rel="noreferrer">https://www.coursera.org/learn/machine-learning/lecture/ND5G5/random-initialization</a>) </p></li>
</ul>

<p><a href="https://i.stack.imgur.com/U0Ppm.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/U0Ppm.jpg" alt="enter image description here"></a>
If we initialize all weights to the same value (e.g. zero or one). In this case, each hidden unit will get exactly the same signal. E.g. if all weights are initialized to 1, each unit gets signal equal to sum of inputs (and outputs sigmoid(sum(inputs))). If all weights are zeros, which is even worse, every hidden unit will get zero signal. No matter what was the input - if all weights are the same, all units in hidden layer will be the same too. This is why one should initialize weights randomly.</p>
Understanding influence of random start weights on neural network performance
<p>Using R and the package <code>neuralnet</code>, I try to model data that looks like this:</p>

<p><a href="https://i.stack.imgur.com/m4Seq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m4Seq.png" alt="training data"></a></p>

<p>These are temperature readings in 10 min intervals over several days (above is a 2 day cutout). Using the code below, I fit a neural network to the data. There are probably simpler ways to model this exact data, but in the future the data might look quite different. Using a single hidden layer with 2 neurons gives me satisfactory results:</p>

<p><a href="https://i.stack.imgur.com/D67DZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/D67DZ.png" alt="good neural network performance"></a></p>

<p>This also works <em>most of the time</em> with more layers and neurons. However, with one hidden layer with one neuron and <em>occasionally</em> with two layers (in my case 3 and 2 neurons respectively), I get rather poor results, always in the same shape:</p>

<p><a href="https://i.stack.imgur.com/FHCA7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FHCA7.png" alt="poor neural network performance"></a></p>

<p>The only thing random is the initialization of start weights, so I assume it's related to that. However, I must admit that I have not fully grasped the theory of neural networks yet. What I would like to know is, whether the poor results are due to a local minimum ('neuralnet' uses resilient backpropagation with weight backtracking by default) and I'm simply out of luck, or if I can avoid such a scenario. I am under the impression that there is an optimal number of hidden nodes for fitting e.g. polynomials of degree 2, 5, 10. If not, what's my best course of action? A larger learning rate? Smaller error threshold? Thanks in advance.</p>

<p>I have <em>not</em> tried tuning the rprop parameters yet, so the solution might lie there.</p>

<p>Code: </p>

<pre><code># DATA ----------------------
minute &lt;- seq(0, 6*24 - 1)
temp &lt;- rep.int(17, 6*24)
temp[(6*7):(6*20)] &lt;- 20
n &lt;- 10
dta &lt;- data.frame(Zeit = minute, Status = temp)
dta &lt;- dta[rep(seq_len(nrow(dta)), n), ]
# Scale everything
maxs &lt;- apply(dta, 2, max) 
mins &lt;- apply(dta, 2, min)

nnInput &lt;- data.frame(Zeit = dta$Zeit, Status = dta$Status)
nnInput &lt;- as.data.frame(scale(nnInput, center = mins, scale = maxs - mins))
trainingData &lt;- nnInput[seq(1, nrow(nnInput), 2), ]
testData     &lt;- nnInput[seq(2, nrow(nnInput), 2), ]

# MODEL ---------------------
model &lt;- as.formula("Status ~ Zeit")
net &lt;- neuralnet::neuralnet(model, 
                            trainingData, 
                            hidden = 2, 
                            threshold = 0.01,
                            linear.output = TRUE,
                            lifesign = "full",
                            stepmax = 100000,
                            rep = 1)

net.results &lt;- neuralnet::compute(net, testData$Zeit)

results &lt;- net.results$net.result * (maxs["Status"] - mins["Status"]) + mins["Status"]
testData &lt;- as.data.frame(t(t(testData) * (maxs - mins) + mins))

cleanOutput &lt;- data.frame(Actual = testData$Status, 
                          Prediction = results, 
                          diff = abs(results - testData$Status))

summary(cleanOutput)

plot(cleanOutput$Actual[1:144], main = "Zeittabelle", xlab = paste("Min. seit 0:00 *", n), ylab = "Temperatur")
lines(cleanOutput$Prediction[1:144], col = "red", lwd = 3)
</code></pre>