<p>I know this is an old post but for anyone coming back to this:</p>
<p>Adding to what Hai-Anh Trinh said, Transformers aren't 'bi-directional', it would be better to call them &quot;omni-directional&quot;. Because of their self-attention method, they are able to consider every single word at the same time, simultaneously.</p>
<p>BERT on the other hand is &quot;deeply bidirectional&quot;. This is because of the masked language model(MLM) pre-training objective that is used in BERT. (there are a lot of resources online, I can link some if need be)</p>
<p>It's easy to get confused so don't worry about it.</p>
<p>(<a href="https://arxiv.org/pdf/1810.04805.pdf;" rel="nofollow noreferrer">https://arxiv.org/pdf/1810.04805.pdf;</a> link to the original BERT paper)
(<a href="https://arxiv.org/pdf/1706.03762.pdf;" rel="nofollow noreferrer">https://arxiv.org/pdf/1706.03762.pdf;</a> link to the original Transformer paper)</p>
How Transformer is Bidirectional - Machine Learning
<p>I am coming from Google BERT context (Bidirectional Encoder representations from Transformers). I have gone through architecture and codes. People say this is <strong>bidirectional <em>by nature</em></strong>. To make it unidirectional attention some mask is to be applied.  </p>

<p>Basically a transformer takes key, values and queries as input; uses encoder decoder architecture; and applies attention to these keys, queries and values. What I understood is we need to pass tokens explicitly rather than transformer understanding this by nature.  </p>

<p>Can someone please explain <strong>what makes transformer is bidirectional by nature</strong></p>