for all but the first embedding matrix and that all embedding vectors $\textbf{e}_{j}$ have the same *base dimension* $\bar{d} := d_0$. Therefore, models based on a mixed dimension embedding layer should be sized with respect to $\bar{d}$. We illustrate the matrix architecture of the mixed dimension embedding layer with two blocks on Fig. \[fig:emb\_layer\], where the parameter budget (total area) consumed by uniform and mixed dimension matrices is the same, but allocated differently.

[0.68]{} ![Matrix Architecture for Uniform and Mixed Dimension Embedding Layers []{data-label="fig:emb_layer"}](images/mixed_arch_9_19.png "fig:"){width="100.00000%"}

Let us now focus on how to find the block structure in the mixed dimension architecture. This includes the row count $n_i$ as well as dimension $d_i$ assigned to each block in the mixed dimension embedding layer. We restrict ourselves to use of *popularity* information for sizing the mixed dimension embedding layer (i.e. frequency $\textbf{f}$ of access of a particular feature; assumed here to be mostly consistent between training and test samples). We note that in principle, one could also use a related but distinct notion of *importance*, that refers to how statistically informative a particular feature is, on average, to the inference of the target variable. Importance could be determined either by domain experts 