<p>In my understanding, the authors of <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">original paper</a> avoid mixing base theory with implementation details. Thus, they defined attention/context size equal to the encoder hidden size (<a href="https://i.stack.imgur.com/i9E98.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/i9E98.gif" alt="enter image description here"></a> for bi-directional LSTM) as below:</p>

<p><a href="https://i.stack.imgur.com/EhBzy.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EhBzy.gif" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/p3Vrh.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/p3Vrh.gif" alt="enter image description here"></a></p>

<p>However, if encoder hidden size is too large, computing attention over long sequences may consume significant amount of time and memory.</p>

<p>So, <a href="https://github.com/tensorflow/tensorflow/blob/283f03c825312efd3319cb37dc1a412288a536ec/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py" rel="nofollow noreferrer">tensorflow implementation</a> introduced additional dense attention_layer with tunable <strong>attention_size</strong> option (<strong>attention_layer_size</strong> in later versions) as below:</p>

<pre><code>  if attention_layer is not None:
    attention = attention_layer(array_ops.concat([cell_output, context], 1))
  else:
    attention = context
</code></pre>

<p><strong>TL;DR;</strong> You can use <strong>attention_size</strong> option to reduce memory consumption of attention mechanism, when encoder hidden size is too large.</p>
what the argument attention_size of tf.contrib.seq2seq.AttentionWrapper mean?
<p>There has an argument <code>anttention_size</code> in<code>tf.contrib.seq2seq.AttentionWrapper</code>, the document says "The basic attention wrapper is tf.contrib.seq2seq.AttentionWrapper. This wrapper accepts an RNNCell instance, an instance of AttentionMechanism, and <strong>an attention depth parameter (attention_size)</strong>;", but what is an attention depth? In the Bahdanau and Luong's paper, I find no attention depth at all, and the source code of the attention mechanism I don't understand clearly. Who can tell me the mean of 'attention_size' and the principle, thank you!</p>