<p>Yes, updating only one weight at the time could result in decreasing error value every time but it's usually infeasible to do such updates in practical solutions using NN. Most of today's architectures usually have ~ 10^6 parameters so one epoch for every parameter could last enormously long. Moreover - because of nature of backpropagation - you usually have to compute loads of different derivatives in order to compute derivative with respect to a parameter given - so you will waste a lot of computations when using such approach.</p>

<p>But the phenomenon which you mention has been noticed a long time ago and there are some ways in dealing with it. There are two most common issues connected with it:</p>

<ol>
<li><strong><em>Covariance shift:</em></strong> it's when error and weight updates of a layer given strongly depends on output from previous layer, so when you update it - the results in the next layer might be different. The most common way to deal with this problem right now is <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow"><strong>Batch normalization</strong></a>.</li>
<li><strong><em>Nolinear function vs Linear Differentation:</em></strong> it's quite uncommon when you think about BP but derivative is a linear operator which might generate a lot of problems in gradient descent. The most countintuitive example is the fact that if you multiply your input by a constant then every derivative will also be multiplied by the same number. This may lead to a lot of problems but most of recent methods of learning do a great job in dealing with it. </li>
</ol>

<p>About BPTT I stronly recomend you Geoffrey Hinton course about ANN and especially this <a href="https://www.youtube.com/watch?v=gPdbTIEMQwY" rel="nofollow">video</a>.</p>
Neural Networks back propogation
<p>I have gone through neural networks and have understood the derivation for back propagation almost perfectly(finally!). However, I had a small doubt.
We are updating all the weights simultaneously, so what is the guarantee that they lead to a smaller cost. If the weights are updated one by one, it would definitely lead to a lesser cost and it would be similar to linear regression. But if you update all the weights simultaneously, might we not cross the minima?</p>

<p>Also, do we update the biases like we update the weights after each forward propagation and back propagation of each test case?</p>

<p>Lastly, I have started reading on RNN's. What are some good resources to understand BPTT in RNN's? </p>