<p>This kind of learning curve is perfectly normal in neural network training (or even in <a href="https://en.wikipedia.org/wiki/Learning_curve" rel="nofollow">real life learning</a>). That said, while the general shape of the curve is typical, we can improve on its steepness. In that respect, I suggest that you implement <a href="https://www.youtube.com/watch?v=8yg2mRJx-z4" rel="nofollow">momentum</a> into your training algorithm. If that does not seem to be enough, your next step would be to implement some adaptive learning rate algorithm such as <a href="http://sebastianruder.com/optimizing-gradient-descent/" rel="nofollow">adadelta, adagrad or rmsprop</a>. Finally, a last thing you may want to try is <a href="http://arxiv.org/abs/1502.03167" rel="nofollow">batch normalization</a>.</p>
How can I speed up learning for feed forward, gradient based backpropagation neural networks
<p>I am using tanh as an activation function.
Let's take one problem for example.</p>

<pre><code>XOR Problem:

1   1   0
0   1   1
1   0   1
0   0   0
</code></pre>

<p>When I train my neural network 500 epochs,
results look like this:</p>

<pre><code>1   1   0.001015
0   1   0.955920
1   0   0.956590
0   0   0.001293
</code></pre>

<p>After another 500 epoch:</p>

<pre><code>1   1   0.000428
0   1   0.971866
1   0   0.971468
0   0   0.000525
</code></pre>

<p>Another 500 epoch:</p>

<pre><code>1   1   0.000193
0   1   0.980982
1   0   0.981241
0   0   0.000227
</code></pre>

<p>It seems that the learning is slowing down alot.
My neural network is taking forver to get precise enough for my costom problems.</p>

<p>Is there anyway to speed up the learning after it starts getting slow like that?</p>

<p>Thanks</p>