title: Understanding influence of random start weights on neural network performance
body: <p>Using R and the package <code>neuralnet</code>, I try to model data that looks like this:</p>
  
  <p><a href="https://i.stack.imgur.com/m4Seq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m4Seq.png" alt="training data"></a></p>
  
  <p>These are temperature readings in 10 min intervals over several days (above is a 2 day cutout). Using the code below, I fit a neural network to the data. There are probably simpler ways to model this exact data, but in the future the data might look quite different. Using a single hidden layer with 2 neurons gives me satisfactory results:</p>
  
  <p><a href="https://i.stack.imgur.com/D67DZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/D67DZ.png" alt="good neural network performance"></a></p>
  
  <p>This also works <em>most of the time</em> with more layers and neurons. However, with one hidden layer with one neuron and <em>occasionally</em> with two layers (in my case 3 and 2 neurons respectively), I get rather poor results, always in the same shape:</p>
  
  <p><a href="https://i.stack.imgur.com/FHCA7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FHCA7.png" alt="poor neural network performance"></a></p>
  
  <p>The only thing random is the initialization of start weights, so I assume it's related to that. However, I must admit that I have not fully grasped the theory of neural networks yet. What I would like to know is, whether the poor results are due to a local minimum ('neuralnet' uses resilient backpropagation with weight backtracking by default) and I'm simply out of luck, or if I can avoid such a scenario. I am under the impression that there is an optimal number of hidden nodes for fitting e.g. polynomials of degree 2, 5, 10. If not, what's my best course of action? A larger learning rate? Smaller error threshold? Thanks in advance.</p>
  
  <p>I have <em>not</em> tried tuning the rprop parameters yet, so the solution might lie there.</p>
  
  <p>Code: </p>
  
  <pre><code># DATA ----------------------
  minute &lt;- seq(0, 6*24 - 1)
  temp &lt;- rep.int(17, 6*24)
  temp[(6*7):(6*20)] &lt;- 20
  n &lt;- 10
  dta &lt;- data.frame(Zeit = minute, Status = temp)
  dta &lt;- dta[rep(seq_len(nrow(dta)), n), ]
  # Scale everything
  maxs &lt;- apply(dta, 2, max) 
  mins &lt;- apply(dta, 2, min)
  
  nnInput &lt;- data.frame(Zeit = dta$Zeit, Status = dta$Status)
  nnInput &lt;- as.data.frame(scale(nnInput, center = mins, scale = maxs - mins))
  trainingData &lt;- nnInput[seq(1, nrow(nnInput), 2), ]
  testData     &lt;- nnInput[seq(2, nrow(nnInput), 2), ]
  
  # MODEL ---------------------
  model &lt;- as.formula("Status ~ Zeit")
  net &lt;- neuralnet::neuralnet(model, 
                              trainingData, 
                              hidden = 2, 
                              threshold = 0.01,
                              linear.output = TRUE,
                              lifesign = "full",
                              stepmax = 100000,
                              rep = 1)
  
  net.results &lt;- neuralnet::compute(net, testData$Zeit)
  
  results &lt;- net.results$net.result * (maxs["Status"] - mins["Status"]) + mins["Status"]
  testData &lt;- as.data.frame(t(t(testData) * (maxs - mins) + mins))
  
  cleanOutput &lt;- data.frame(Actual = testData$Status, 
                            Prediction = results, 
                            diff = abs(results - testData$Status))
  
  summary(cleanOutput)
  
  plot(cleanOutput$Actual[1:144], main = "Zeittabelle", xlab = paste("Min. seit 0:00 *", n), ylab = "Temperatur")
  lines(cleanOutput$Prediction[1:144], col = "red", lwd = 3)
  </code></pre>