offline, when they only involve post-training processing, or online, when the compression process is interleaved with or otherwise materially alters the training process. Simple offline compression algorithms include post-training quantization, pruning or low-rank SVD [@Andrews2015; @Bhavana2019; @Sattigeri; @Sun2016]. Model distillation techniques, such as compositional coding [@Shu2017] and neural binarization [@Tissier2018] are also a form of complex offline compression in which autoencoders are trained to mimic uncompressed, pre-trained embedding layers. Online compression algorithms include quantization-aware training, gradual pruning, and periodic regularization [@alvarez2017compression; @Elthakeb2019; @frankle2018lottery; @Naumov2018reg; @park2018value]. We note that many of these compression algorithms are not unique to embedding layers, and are widely used in the model compression literature.

On the other hand, we also have compressed architectures, which attempt to architect embedding representations of comparable statistical quality with fewer parameters. Compressed architectures have the advantage of not only reducing memory requirements for inference time, but also at training time. This is the approach followed by hashing-based and tensor factorization methods [@attenberg2009collaborative; @gao2018cuckoo; @karatzoglou2010collaborative; @Khrulkov2019; @Shi2018], which focus on reducing the number of parameters used in an embedding layer by re-using parameters in various ways. These techniques stand in contrast with our approach, which focuses on non-uniform reduction of the dimensionality 