A major obstacle to using approximate activations for training is that, with the standard implementation of back-propagation (backprop), approximation errors compound across the forward and then backward pass through all layers.
we need to store all intermediate activations during training because they are needed to compute gradients during back-propagation: ( 5)-( 8) involve not just the values of the incoming gradient, but also the values of the activations themselves.