several communication and memory access overheads if learning is executed on hardware whose memory can not accommodate all the network weights and activations. For large scale convolutional networks, this practically includes all CPU and GPU devices where on-chip memory is limited to few tens of MBytes, while state of the art deep convolutional networks typically require several hundred MBytes to several GBytes in order to store the network weights and mini-batch activations [@Rhu_etal16]. Data thus has to be continuously shuttled between the compute device and external memory. This is the case even in custom accelerators developed to accelerate just the inference (feed-forward) phase [@Himavathi_etal07; @Cavigelli_etal15; @Chen_etal16; @Han_etal16; @Ardakani_etal16; @Aimar_etal17; @Jouppi_etal17], where a complete forward pass through a large-scale convolutional network can not be executed completely on the accelerator without having to access external memory to store intermediate activations and to load weights.

Improvements in memory bandwidth significantly lag improvements in computing elements speed [@Wulf_McKee95]. Reducing memory traffic in a compute intensive task such as learning deep networks thus improves performance as it relaxes the requirements on the memory bandwidth and latency needed to keep the compute elements occupied, allowing either the compute elements to run at higher frequencies or the 