that for uniform dimension $d = 4$ (orange line) the model underfits the data, because the attained loss is significantly higher than the one achieved with other hyper-parameters. Also, notice that the model with uniform dimension $d = 16$ (blue line) severely overfits after about $10$ epochs. On the other hand, mixed dimension embeddings (green line) train well and achieve the lowest validation loss among other hyper-parameters.

This evidence that at dimension $d=4$, the popular embedding vectors are underfitting, and that at dimension $d=16$ the unpopular embedding vectors are overfitting. At the $d=4$ parameter budget, the mixed dimension embeddings at $\alpha = 0.6$ use a baseline dimension $\bar{d}=32$ and result in an embedding layer architecture that can more adequately fit both popular and unpopular embedding vectors.

In Fig. \[fig:mf\_test\_loss\_summary\], we report the test loss for different $\alpha$ at varying total parameter budgets using optimal early termination. We can see that using mixed dimension embedding layers generally improves the memory-performance frontier at each parameter budget. We point out a simple trend illustrated by the uniform dimension (blue line): performance improves with number of parameters, until we reach a critical point, after which performance decreases with increasing parameters. Notice that at all 