<p>Attention, as introduced in <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention Is All You Need</a>, IMHO, is quite similar to what our brains use as the attention mechanism. </p>

<p>We have something named LGN in our brains responsible for filtering out unnecessary information for the task at hand. For instance, if I start looking for my keys, my brain is going to pay less attention to objects with a color other than silver or gold (hopefully). Now, I'm not aware of the higher level attention mechanisms of the human brain. However, the one thing that's clear is that information is passing through each layer before the attention and represented in the form of neural activities.</p>

<p>When you feed your artificial model representation of the world's current state, information is going to be represented as tensors and, similarly, attention allows you to see what you need to see to make the best decision (similar to finding keys).</p>
Is attention mechanism really attention or just looking back at memory again?
<p>When reading attention mechanism, I am confusing about the term attention. Is it the same with our attention nature as described in it usual definition?</p>