and achieve on par accuracy with 16$\times$ fewer parameters by using mixed dimension embeddings.*

[0.49]{} ![Accuracy under different parameters budgets[]{data-label="fig:dlrm_performance"}](images/ctr_learning_curve_finalv2.png "fig:"){width="\textwidth"}

[0.49]{} ![Accuracy under different parameters budgets[]{data-label="fig:dlrm_performance"}](images/ctr_rate_performance_9_19.png "fig:"){width="\textwidth"}

To summarize we identify two distinct mechanisms by which mixed dimension embedding layers improve upon uniform-dimension layers in our experiments.

i\) Generalization – At training time, embeddings learn to represent categorical features such as users or products. We find that at a uniform-dimension, frequently-accessed vectors are often under-fitting, whereas infrequently-accessed vectors are often over-fitting. Thus, mixed dimension embeddings actually learn in more parameter-efficient manner.

ii\) Allocation – Learning aside, at inference time, for a sufficiently constrained parameter budget, one faces a resource allocation trade-off. Even with an oracle factorization of the target matrix, under an expected distortion metric at a given parameter budget, it is more efficient to allocate more parameters to frequently-accessed vectors than to infrequently-accessed vectors.

Conclusion
==========

We propose mixed dimension embedding layers in which the dimension of a particular embedding vector is based on its popularity. This approach addresses severe inefficiencies in the number of parameters used in uniform dimension embedding layers. It offers a superior trade-off between number of parameters and statistical performance of the model. In 