illustrates the data traffic and the number of MAC operations needed during standard backpropagation training. The data traffic in Fig. \[fig:traffic\_a\] assumes the compute device has enough on-board memory to buffer the output activations of one layer in order to use these activations to calculate the next layer’s activation. We also assume the compute device does not need the parameters of any layer to be streamed in more than once during each forward pass and during each backward pass. These assumptions would hold true if the accelerator has at least $\max_i (\lvert P^i\rvert +  \lvert A^i\rvert)$ words of on-board memory. During the forward pass, the activations of all layers have to be streamed out to external memory so they can be used in the backward pass. The number of MAC operations needed to calculate the activation of layer $i$ is $R^{i-1}\lvert A^{i-1} \rvert$. During the backward pass, the compute device buffers the back-propagated errors of one layer and uses them to calculate the errors at the preceding layer. $R^{i}\lvert A^{i} \rvert$ MAC operations are needed to calculate the weight updates for $P^{i+1}$. An additional $R^{i}\lvert A^{i} \rvert$ MAC operations are needed to backpropagate the errors from layer $i+1$ to layer 