returns the series of vectors accordingly. We describe the architecture of each block as the following.

The multi-head self-attention layer takes a series of vectors, $X_1, \dots, X_T$. Each vector is projected to latent space by projection matrices $W^{Q}, W^{K}\in \mathbb{R}^{d_{\mathrm{model}} \times d_{K}}$ and $W^{V}\in \mathbb{R}^{d_{\mathrm{model}} \times d_{V}}$: $$\begin{aligned}
\label{eqn:attn-proj}
\begin{split}
Q = [q_1, \dots, q_T]^{T} & = X W^Q \\
K = [k_1, \dots, k_T]^{T} & = X W^K \\
V = [v_1, \dots, v_T]^{T} & = X W^V
\end{split}\end{aligned}$$ Here $X = [X_1, \dots, X_T]^{T}$ and each $q_i$, $k_i$ and $v_i$ are the query, key and value of $X_i$ respectively. The output of the self-attention is then obtained as weighted sum of values with coefficients determined by the dot products between queries and keys: $$\begin{aligned}
\label{eqn:attn-softmax}
\mathrm{Attention}(X) = \mathrm{Softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right) V\end{aligned}$$ Models with self-attention layers often use multiple heads to jointly attend information from different representative subspaces. Following this, we apply attention $h$ times to the same query-key-value entries with different projection matrices for output: $$\begin{aligned}
\label{eqn:multihead}
\mathrm{Multihead}(X) = \mathrm{concat}(\mathrm{head}_{1},  \dots ,\mathrm{head}_{T}) W^O\end{aligned}$$ Here, each $\mathrm{head}_{i}$ is equal to the output of self-attention in Equation \[eqn:attn-softmax\] with corresponding projection matrices $W_i^Q$, $W_i^K$ and $W_i^V$ in Equation \[eqn:attn-proj\]. We use 