<p>If you can train two networks, to the same accuracy, but one of them only needs to process half as much data, then yes that is a good thing.</p>

<p>The resulting network will not be any faster to execute during inference time, but there are still several important benefits to the training process.</p>

<ul>
<li>Training will take half as long. This is valuable by itself. It is extra valuable when you consider that you can now try twice as many ideas in the same amount of time. That will improve results quality for the entire process.</li>
<li>Faster convergence can reduce generalization error and overfitting. The optimization does not have as many opportunities to "fidget" and find opportunities to overfit.</li>
<li>Extremely fast convergence, called <a href="https://arxiv.org/abs/1708.07120" rel="nofollow noreferrer">super-convergence</a>, can improve the final <em>training</em> error while still keeping generalization error low, leading to better validation scores too.</li>
</ul>

<p>Speaking more generally, there is a lot of research and other activity on the topic of how to make networks train as quickly and cheaply as possible. One such benchmark is <a href="https://dawn.cs.stanford.edu/benchmark/" rel="nofollow noreferrer">DAWNBench</a>, which sets a target accuracy to achieve and then ranks approaches based on how fast they reach that target, and how much the GPUs or other infrastructure cost to do it.</p>

<p>This general idea of "cost reduction" is also one of the drivers behind the general idea of <a href="https://en.wikipedia.org/wiki/Transfer_learning" rel="nofollow noreferrer">Transfer Learning</a>.</p>
Can reducing the number of back propagation steps improve training performance?
<p>I want to know how beneficial it would be if we could reduce the number of back propagation steps by 50%.</p>

<p>For example, let's say a neural network performed back propagation 1000 times for training. And another neural network performs back propagation 500 to get trained (Lets assume that both of them gave same accuracy after training). Will the second one be significantly faster? Or does it not matter much? It will increase the speed of training.</p>