

[Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]









What’s Wrong with Seq2Seq Model?





encoder-decoder model with additive attention layer



Born for Translation



  • encoder hidden states;
  • decoder hidden states;
  • alignment between source and target.

encoder-decoder model with additive attention layer



Now let’s define the attention mechanism introduced in NMT in a scientific way. Say, we have a source sequence \(\mathbf{x}\) of length \(n\) and try to output a target sequence \(\mathbf{y}\) of length \(m\):

\[\begin{aligned} \mathbf{x} &= [x_1, x_2, \dots, x_n] \\ \mathbf{y} &= [y_1, y_2, \dots, y_m] \end{aligned}\]


The encoder is a bidirectional RNN (or other recurrent network setting of your choice) with a forward hidden state \(\overrightarrow{\boldsymbol{h}}_i\) and a backward one \(\overleftarrow{\boldsymbol{h}}_i\). A simple concatenation of two represents the encoder state. The motivation is to include both the preceding and following words in the annotation of one word.

\[\boldsymbol{h}_i = [\overrightarrow{\boldsymbol{h}}_i^\top; \overleftarrow{\boldsymbol{h}}_i^\top]^\top, i=1,\dots,n\]

The decoder network has hidden state \(\boldsymbol{s}_t=f(\boldsymbol{s}_{t-1}, y_{t-1}, \mathbf{c}_t)\) for the output word at position t, \(t=1,\dots,m\), where the context vector \(\mathbf{c}_t\) is a sum of hidden states of the input sequence, weighted by alignment scores:

\[\begin{aligned} \mathbf{c}_t &= \sum_{i=1}^n \alpha_{t,i} \boldsymbol{h}_i & \small{\text{; Context vector for output }y_t}\\ \alpha_{t,i} &= \text{align}(y_t, x_i) & \small{\text{; How well two words }y_t\text{ and }x_i\text{ are aligned.}}\\ &= \frac{\exp(\text{score}(\boldsymbol{s}_{t-1}, \boldsymbol{h}_i))}{\sum_{i'=1}^n \exp(\text{score}(\boldsymbol{s}_{t-1}, \boldsymbol{h}_{i'}))} & \small{\text{; Softmax of some predefined alignment score.}}. \end{aligned}\]

The alignment model assigns a score \(\alpha_{t,i}\) to the pair of input at position i and output at position t, \((y_t, x_i)\), based on how well they match. The set of \(\{\alpha_{t, i}\}\) are weights defining how much of each source hidden state should be considered for each output. In Bahdanau’s paper, the alignment score \(\alpha\) is parametrized by a feed-forward network with a single hidden layer and this network is jointly trained with other parts of the model. The score function is therefore in the following form, given that tanh is used as the non-linear activation function:

\[\text{score}(\boldsymbol{s}_t, \boldsymbol{h}_i) = \mathbf{v}_a^\top \tanh(\mathbf{W}_a[\boldsymbol{s}_t; \boldsymbol{h}_i])\]

where both \(\mathbf{v}_a\) and \(\mathbf{W}_a\) are weight matrices to be learned in the alignment model.


alignment matrix



A Family of Attention Mechanisms




Name Alignment score function Citation
Content-base attention \(\text{score}(\boldsymbol{s}_t, \boldsymbol{h}_i) = \text{cosine}[\boldsymbol{s}_t, \boldsymbol{h}_i]\) Graves2014
Location-Base \(\alpha_{t,i} = \text{softmax}(\mathbf{W}_a \boldsymbol{s}_t)\)
General \(\text{score}(\boldsymbol{s}_t, \boldsymbol{h}_i) = \boldsymbol{s}_t^\top\mathbf{W}_a\boldsymbol{h}_i\)
where \(\mathbf{W}_a\) is a trainable weight matrix in the attention layer.
Dot-Product \(\text{score}(\boldsymbol{s}_t, \boldsymbol{h}_i) = \boldsymbol{s}_t^\top\boldsymbol{h}_i\) Luong2015
Scaled Dot-Product(^) \(\text{score}(\boldsymbol{s}_t, \boldsymbol{h}_i) = \frac{\boldsymbol{s}_t^\top\boldsymbol{h}_i}{\sqrt{n}}\)

(^) It adds a scaling factor \(1/\sqrt{n}\), motivated by the concern when the input is large, the softmax function may have an extremely small gradient, hard for efficient learning.

Here are a summary of broader categories of attention mechanisms:

Name Definition Citation
Global/Soft Attending to the entire input state space. Xu2015







Soft vs Hard Attention





    • Pro: the model is smooth and differentiable.
    • Con: expensive when the source input is large.
    • Pro: less calculation at the inference time.

Global vs Local Attention




Neural Turing Machines





NTM contains two major components, a controller neural network and a memory bank. Controller: is in charge of executing operations on the memory. It can be any type of neural network, feed-forward or recurrent. Memory: stores processed information. It is a matrix of size \(N \times M\), containing N vector rows and each has \(M\) dimensions.



Fig 10. Neural Turing Machine Architecture.

Reading and Writing

When reading from the memory at time t, an attention vector of size \(N\), \(\mathbf{w}_t\) controls how much attention to assign to different memory locations (matrix rows). The read vector \(\mathbf{r}_t\) is a sum weighted by attention intensity:

\[\mathbf{r}_t = \sum_{i=1}^N w_t(i)\mathbf{M}_t(i)\text{, where }\sum_{i=1}^N w_t(i)=1, \forall i: 0 \leq w_t(i) \leq 1\]

where \(w_t(i)\) is the \(i\)-th element in \(\mathbf{w}_t\) and \(\mathbf{M}_t(i)\) is the \(i\)-th row vector in the memory.

When writing into the memory at time t, as inspired by the input and forget gates in LSTM, a write head first wipes off some old content according to an erase vector \(\mathbf{e}_t\) and then adds new information by an add vector \(\mathbf{a}_t\).

\[\begin{aligned} \tilde{\mathbf{M}}_t(i) &= \mathbf{M}_{t-1}(i) [\mathbf{1} - w_t(i)\mathbf{e}_t] &\scriptstyle{\text{; erase}}\\ \mathbf{M}_t(i) &= \tilde{\mathbf{M}}_t(i) + w_t(i) \mathbf{a}_t &\scriptstyle{\text{; add}} \end{aligned}\]

Attention Mechanisms

In Neural Turing Machine, how to generate the attention distribution \(\mathbf{w}_t\) depends on the addressing mechanisms: NTM uses a mixture of content-based and location-based addressings.

Content-based addressing

The content-addressing creates attention vectors based on the similarity between the key vector \(\mathbf{k}_t\) extracted by the controller from the input and memory rows. The content-based attention scores are computed as cosine similarity and then normalized by softmax. In addition, NTM adds a strength multiplier \(\beta_t\) to amplify or attenuate the focus of the distribution.

\[w_t^c(i) = \text{softmax}(\beta_t \cdot \text{cosine}[\mathbf{k}_t, \mathbf{M}_t(i)]) = \frac{\exp(\beta_t \frac{\mathbf{k}_t \cdot \mathbf{M}_t(i)}{\|\mathbf{k}_t\| \cdot \|\mathbf{M}_t(i)\|})}{\sum_{j=1}^N \exp(\beta_t \frac{\mathbf{k}_t \cdot \mathbf{M}_t(j)}{\|\mathbf{k}_t\| \cdot \|\mathbf{M}_t(j)\|})}\]


Then an interpolation gate scalar \(g_t\) is used to blend the newly generated content-based attention vector with the attention weights in the last time step:

\[\mathbf{w}_t^g = g_t \mathbf{w}_t^c + (1 - g_t) \mathbf{w}_{t-1}\]

Location-based addressing

The location-based addressing sums up the values at different positions in the attention vector, weighted by a weighting distribution over allowable integer shifts. It is equivalent to a 1-d convolution with a kernel \(\mathbf{s}_t(.)\), a function of the position offset. There are multiple ways to define this distribution. See Fig. 11. for inspiration.


Fig. 11. Two ways to represent the shift weighting distribution \(\mathbf{s}_t\).

Finally the attention distribution is enhanced by a sharpening scalar \(\gamma_t \geq 1\).

\[\begin{aligned} \tilde{w}_t(i) &= \sum_{j=1}^N w_t^g(j) s_t(i-j) & \scriptstyle{\text{; circular convolution}}\\ w_t(i) &= \frac{\tilde{w}_t(i)^{\gamma_t}}{\sum_{j=1}^N \tilde{w}_t(j)^{\gamma_t}} & \scriptstyle{\text{; sharpen}} \end{aligned}\]

The complete process of generating the attention vector \(\mathbf{w}_t\) at time step t is illustrated in Fig. 12. All the parameters produced by the controller are unique for each head. If there are multiple read and write heads in parallel, the controller would output multiple sets.



Pointer Network


pointer network


The Ptr-Net outputs a sequence of integer indices, \(\boldsymbol{c} = (c_1, \dots, c_m)\) given a sequence of input vectors \(\boldsymbol{x} = (x_1, \dots, x_n)\) and \(1 \leq c_i \leq n\). The model still embraces an encoder-decoder framework. The encoder and decoder hidden states are denoted as \((\boldsymbol{h}_1, \dots, \boldsymbol{h}_n)\) and \((\boldsymbol{s}_1, \dots, \boldsymbol{s}_m)\), respectively. Note that \(\mathbf{s}_i\) is the output gate after cell activation in the decoder. The Ptr-Net applies additive attention between states and then normalizes it by softmax to model the output conditional probability:

\[\begin{aligned} y_i &= p(c_i \vert c_1, \dots, c_{i-1}, \boldsymbol{x}) \\ &= \text{softmax}(\text{score}(\boldsymbol{s}_t; \boldsymbol{h}_i)) = \text{softmax}(\mathbf{v}_a^\top \tanh(\mathbf{W}_a[\boldsymbol{s}_t; \boldsymbol{h}_i])) \end{aligned}\]




The secret recipe is carried in its model architecture.

Key, Value and Query

The major component in the transformer is the unit of multi-head self-attention mechanism. The transformer views the encoded representation of the input as a set of key-value pairs, \((\mathbf{K}, \mathbf{V})\), both of dimension \(n\) (input sequence length); in the context of NMT, both the keys and values are the encoder hidden states. In the decoder, the previous output is compressed into a query (\(\mathbf{Q}\) of dimension \(m\)) and the next output is produced by mapping this query and the set of keys and values.


\[\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{n}})\mathbf{V}\]

Multi-Head Self-Attention

multi-head scaled dot-product attention



\[\begin{aligned} \text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) &= [\text{head}_1; \dots; \text{head}_h]\mathbf{W}^O \\ \text{where head}_i &= \text{Attention}(\mathbf{Q}\mathbf{W}^Q_i, \mathbf{K}\mathbf{W}^K_i, \mathbf{V}\mathbf{W}^V_i) \end{aligned}\]

where \(\mathbf{W}^Q_i\), \(\mathbf{W}^K_i\), \(\mathbf{W}^V_i\), and \(\mathbf{W}^O\) are parameter matrices to be learned.


Transformer encoder



  • A stack of N=6 identical layers.
  • Each sub-layer adopts a residual connection and a layer normalization. All the sub-layers output data of the same dimension \(d_\text{model} = 512\).


Transformer decoder


The decoder is able to retrieval from the encoded representation.

  • A stack of N = 6 identical layers

Full Architecture

Finally here is the complete view of the transformer’s architecture:

  • Both the source and target sequences first go through embedding layers to produce data of the same dimension \(d_\text{model} =512\).

Transformer model









Self-Attention GAN




Conv vs self-attention on images


The SAGAN adopts the non-local neural network to apply the attention computation. The convolutional image feature maps \(\mathbf{x}\) is branched out into three copies, corresponding to the concepts of key, value, and query in the transformer:

  • Key: \(f(\mathbf{x}) = \mathbf{W}_f \mathbf{x}\)
  • Query: \(g(\mathbf{x}) = \mathbf{W}_g \mathbf{x}\)
  • Value: \(h(\mathbf{x}) = \mathbf{W}_h \mathbf{x}\)


\[\begin{aligned} \alpha_{i,j} &= \text{softmax}(f(\mathbf{x}_i)^\top g(\mathbf{x}_j)) \\ \mathbf{o}_j &= \mathbf{W}_v \Big( \sum_{i=1}^N \alpha_{i,j} h(\mathbf{x}_i) \Big) \end{aligned}\]



Note that \(\alpha_{i,j}\) is one entry in the attention map, indicating how much attention the model should pay to the \(i\)-th position when synthesizing the \(j\)-th location. \(\mathbf{W}_f\), \(\mathbf{W}_g\), and \(\mathbf{W}_h\) are all 1x1 convolution filters. If you feel that 1x1 conv sounds like a weird concept (i.e., isn’t it just to multiply the whole feature map with one number?), watch this short tutorial by Andrew Ng. The output \(\mathbf{o}_j\) is a column vector of the final output \(\mathbf{o}= (\mathbf{o}_1, \mathbf{o}_2, \dots, \mathbf{o}_j, \dots, \mathbf{o}_N)\).


\[\mathbf{y} = \mathbf{x}_i + \gamma \mathbf{o}_i\]

While the scaling parameter \(\gamma\) is increased gradually from 0 during the training, the network is configured to first rely on the cues in the local regions and then gradually learn to assign more weight to the regions that are further away.

SAGAN examples


Cited as:

  title   = "Attention? Attention!",
  author  = "Weng, Lilian",
  journal = "",
  year    = "2018",
  url     = ""


See you in the next post :D



[2] “Neural Machine Translation (seq2seq) Tutorial”












