However, we need to store all intermediate activations during training because they are needed to compute gradients during back-propagation: ( 5)-( 8) involve not just the values of the incoming gradient, but also the values of the activations themselves.
training requires enough available memory to hold the activations of all layers in the network.