A number of works focus on reducing the memory footprint of a model during inference, e.g., by compression [7] and quantization [11], to ensure that it can be deployed on resource-limited mobile devices, while still delivering reasonable accuracy.
these methods do not address the significant amount of memory required to train the networks themselves (note [7,11] require storing full-precision activations during training), which is significantly larger than what is needed for inference.