<p>Basically - initialization is really important. If you don't initialize it randomly then you might make your network not working at all (e.g. by setting all the weights to <code>0</code>). It is also proven that for <a href="http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2010_GlorotB10.pdf">sigmoid</a> and <a href="http://arxiv.org/pdf/1502.01852v1.pdf">relu</a> a certain kind of activation might help in training your network.</p>

<p>But in your case - I think that the differences are mostly made by the complexity of your problem. With a models with a complexity which seem to fit the complexity of your problem performs nice. The other models may suffer for the following reasons:</p>

<ol>
<li><strong>Too small complexity</strong> - with one node maybe you are basically not able to learn the proper function.</li>
<li><strong>Too big complexity</strong> - with two-layer network you might experience stucking in a local minimas. Increasing the number of parameters of your network is also increasing the size of parameter space. Of course - one hand you might get the better model - on the other hand - you may land in this region of a parameter space which will result in poor solution. Maybe trying the same model with different initialization - and choosing the best model might overcome this issue.</li>
</ol>

<p><strong>UPDATE:</strong></p>

<ol>
<li><p>With small network sizes - it is quite usual to stuck in a local minimum. Depending on the amount of time which you need to train your network you may use the following techniques to overcome that:</p>

<ul>
<li><strong>Dropout / Batch normalization / Batch learning randomization :</strong> when you are able to train your network for a little bit longer time - you might use a randomization properties of dropout or batch normalization. Due to this random fluctuations you are able to move from poor local minima (which are usually believed to be relatively shallow).</li>
<li><strong>Cross - validation / Multiple run:</strong> When you are starting your training multiple times - the probability that you will finish in a poor minimum significantly decreases.</li>
</ul></li>
<li><p>About the connection between layer size and polynomial degree - I think that the question is not clearly stated. You must specify more details like e.g. the activation function. I also think that the nature of a polynomials and functions which could be modelled by a classic neural networks differs a lot. In polynomials - the small change in parameters values usually tends to much higher difference than in neural network case. Usually a derivative of a neural network is a bounded function whereas the polynomial derivative is unbounded when the degree is bigger that 2. Due to this facts I think - that looking for a dependency between a polynomial degree and a size of a hidden layer might be not worth serious considerations.</p></li>
</ol>
Understanding influence of random start weights on neural network performance
<p>Using R and the package <code>neuralnet</code>, I try to model data that looks like this:</p>

<p><a href="https://i.stack.imgur.com/m4Seq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m4Seq.png" alt="training data"></a></p>

<p>These are temperature readings in 10 min intervals over several days (above is a 2 day cutout). Using the code below, I fit a neural network to the data. There are probably simpler ways to model this exact data, but in the future the data might look quite different. Using a single hidden layer with 2 neurons gives me satisfactory results:</p>

<p><a href="https://i.stack.imgur.com/D67DZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/D67DZ.png" alt="good neural network performance"></a></p>

<p>This also works <em>most of the time</em> with more layers and neurons. However, with one hidden layer with one neuron and <em>occasionally</em> with two layers (in my case 3 and 2 neurons respectively), I get rather poor results, always in the same shape:</p>

<p><a href="https://i.stack.imgur.com/FHCA7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FHCA7.png" alt="poor neural network performance"></a></p>

<p>The only thing random is the initialization of start weights, so I assume it's related to that. However, I must admit that I have not fully grasped the theory of neural networks yet. What I would like to know is, whether the poor results are due to a local minimum ('neuralnet' uses resilient backpropagation with weight backtracking by default) and I'm simply out of luck, or if I can avoid such a scenario. I am under the impression that there is an optimal number of hidden nodes for fitting e.g. polynomials of degree 2, 5, 10. If not, what's my best course of action? A larger learning rate? Smaller error threshold? Thanks in advance.</p>

<p>I have <em>not</em> tried tuning the rprop parameters yet, so the solution might lie there.</p>

<p>Code: </p>

<pre><code># DATA ----------------------
minute &lt;- seq(0, 6*24 - 1)
temp &lt;- rep.int(17, 6*24)
temp[(6*7):(6*20)] &lt;- 20
n &lt;- 10
dta &lt;- data.frame(Zeit = minute, Status = temp)
dta &lt;- dta[rep(seq_len(nrow(dta)), n), ]
# Scale everything
maxs &lt;- apply(dta, 2, max) 
mins &lt;- apply(dta, 2, min)

nnInput &lt;- data.frame(Zeit = dta$Zeit, Status = dta$Status)
nnInput &lt;- as.data.frame(scale(nnInput, center = mins, scale = maxs - mins))
trainingData &lt;- nnInput[seq(1, nrow(nnInput), 2), ]
testData     &lt;- nnInput[seq(2, nrow(nnInput), 2), ]

# MODEL ---------------------
model &lt;- as.formula("Status ~ Zeit")
net &lt;- neuralnet::neuralnet(model, 
                            trainingData, 
                            hidden = 2, 
                            threshold = 0.01,
                            linear.output = TRUE,
                            lifesign = "full",
                            stepmax = 100000,
                            rep = 1)

net.results &lt;- neuralnet::compute(net, testData$Zeit)

results &lt;- net.results$net.result * (maxs["Status"] - mins["Status"]) + mins["Status"]
testData &lt;- as.data.frame(t(t(testData) * (maxs - mins) + mins))

cleanOutput &lt;- data.frame(Actual = testData$Status, 
                          Prediction = results, 
                          diff = abs(results - testData$Status))

summary(cleanOutput)

plot(cleanOutput$Actual[1:144], main = "Zeittabelle", xlab = paste("Min. seit 0:00 *", n), ylab = "Temperatur")
lines(cleanOutput$Prediction[1:144], col = "red", lwd = 3)
</code></pre>