CF and CTR prediction tasks.

Let a mixed dimension embedding layer $\mathbf{\bar{E}}$ consist of $k+1$ blocks and be defined by $2k+1$ matrices, so that $$\mathbf{\bar{E}} = (\bar{E}^{(0)}, \bar{E}^{(1)},...,\bar{E}^{(k)}, P^{(1)},...,P^{(k)})$$

with $\bar{E}^{(i)} \in \mathbb{R}^{n_i \times d_i}$ and $P^{(i)} \in \mathbb{R}^{d_i \times d_0}$ for $i=0,...,k$, where $P^{(0)} \in \mathbb{R}^{d_0 \times d_0}$ is implicitly defined as identity.

Let us assume that the dimensions of these blocks are fixed. Then, forward propagation for a mixed dimension embedding layer takes an index $x$ in the range from $1$ to $n = \sum_{i=0}^k n_i$, and produces an embedding vector $\textbf{e}_x$ as defined in Alg. \[alg:emb\_layer\]. The steps involved in this algorithm are differentiable, therefore we can perform backward propagation through this layer and update matrices $\bar{E}^{(i)}$ and $P^{(i)}$ accordingly during training. We note that Alg. \[alg:emb\_layer\] may be generalized to support multi-hot lookups, where embedding vectors corresponding to some $z$ query indices are fetched and reduced by a differentiable operator, such as add, multiply or concatenation.

**Input:** Index $x \in [n]$ **Output:** Embedding vector $\textbf{e}_{x}$ $i \gets 0$ and $t \gets 0$ $t \gets t + n_i$ $i \gets i + 1$ $\textbf{e}_{x} \gets \bar{E}^{(i)}[x-t]P^{(i)}$ **if** $i>0$ **else** $\bar{E}^{(i)}[x-t]$

\[alg:emb\_layer\]

Note that we return projected embeddings 