<p>In <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">Neural Machine Translation by Jointly Learning to Align and Translate</a> they give a description of the (Bahdanau) attention mechanism; essentially what happens is that you compute scalar "alignment scores" <code>a_1, a_2, ..., a_n</code> that indicate how important each element of your encoded input sequence is at a given moment in time (i.e. which part of the input sentence you should pay attention to right now in the current timestep).</p>

<p>Assuming your (encoded) input sequence that you want to "pay attention"/"attend over" is a sequence of vectors denoted as <code>e_1, e_2, ..., e_n</code>, the context vector at a given timestep is the weighted sum over all of these as determined by your alignment scores:</p>

<p><code>context = c := (a_1*e_1) + (a_2*e_2) + ... + (a_n*e_n)</code></p>

<p>(Remember that the <code>a_k</code>'s are scalars; you can think of this as an "averaged-out" letter/word in your sentence --- so ideally if your model is trained well, the context looks most similar to the <code>e_i</code> you want to pay attention to the most, but bears a little bit of resemblance to <code>e_{i-1}</code>, <code>e_{i+1}</code>, etc. Intuitively, think of a "smeared-out" input element, if that makes any sense...)</p>

<p>Anyway, if <code>attention_layer_size</code> is not <code>None</code>, then it specifies the number of hidden units in a feedforward layer within your decoder that is used to mix this context vector with the output of the decoder's internal RNN cell to get the attention value. If <code>attention_layer_size == None</code>, it just uses the context vector above as the attention value, and no mixing of the internal RNN cell's output is done. (When I say "mixing", I mean that the context vector and the RNN cell's output are concatenated and then projected to the dimensionality that you specify by setting <code>attention_layer_size</code>.)</p>

<p>The relevant part of the implementation is at <a href="https://github.com/tensorflow/tensorflow/blob/r1.2/tensorflow/contrib/seq2seq/python/ops/attention_wrapper.py#L731" rel="nofollow noreferrer">this line</a> and has a description of how it's computed.</p>

<p>Hope that helps!</p>
Clarification of TensorFlow AttentionWrapper's Layer Size
<p>In <code>tensorflow.contrib.seq2seq</code>'s <code>AttentionWrapper</code>, what does "depth" refer to as stated in the <code>attention_layer_size</code> documentation? When the documentation says to "use the context as attention" if the value is <code>None</code>, what is meant by "the context"?</p>