The cost function is minimised to zero when the weights propagate the input in such a way that the network output vector equals the desired output vector.

Since the weights are continuous variables, the numerical partial derivatives of the cost function w.r.t. each weight can be found by approximating $\frac{\partial C}{\partial w}\approx \frac{ C(w+\epsilon)-C(w)}{ \epsilon}$. After each propagation, these partial derivatives are computed and the weights are altered in the direction of greatest decrease of the cost function. Specifically, each weight $w^{(l)}_{jk}$ is increased by $\mathrm{\delta} w^{(l)}_{jk}$, with $$\label{eq:learningrule}
\mathrm{\delta} w^{(l)}_{jk}=-\eta\frac{\partial C}{\partial w^{(l)}_{jk}},$$ where $\eta$ is an adjustable non-negative parameter. This training procedure is known as gradient descent [@Nielsen15].

Note that gradient descent normally also requires a continuous and differentiable activation function, to allow small changes in the weights to relate to small changes in the cost. For this reason, the Heaviside activation function has traditionally been replaced by a sigmoid function [@Nielsen15; @Azoff94]. Nevertheless, gradient descent has also been achieved using Heaviside activation functions, by taking the weights as Gaussian variables and taking partial derivatives w.r.t. the means and standard deviations of the appropriate Gaussian distributions [@BartlettDowns92; @DownsGaynier95].

In the reversible generalisation, where each neuron is replaced by a 