<p>It is bidirectional because it uses context from both sides of the current word (instead of e.g. using just the previous few words it uses the whole sequence).</p>
<p>It depends on how much you want to go into detail but basically there are the attention and the self-attention mechanisms to make this &quot;handle everything in the sequence at once&quot; way work.</p>
<p>In a nutshell the attention mechanism means that instead of going through the sentence sequentially/word-by-word, the entire sequence is used to do the decoding on the currently handled word while using an attention system to give weights to decide which word in the input gets how much say in how the current word is handled.</p>
<p>The Self-Attention mechanism means that even for the encoding of the input sequence itself the context (rest of the sentence) is already used. So e.g. if you have a sentence with an &quot;it&quot; that is used as a pronoun, the encoding of that token is going to be strongly context dependent. Self-Attention means similarily to attention there is a weighting function for which other input token is how relevant for the encoding of the current input tokens.</p>
<p>A popular way to explain Self-Attention is this: <code>The cat ran over the street, because it got startled.</code> The encoding of <code>it</code> in this sentence is strongly dependent on <code>The cat</code> and a bit dependent on <code>the street</code>, because the model learnt during the pre-training that to predict masked words after/around <code>it</code> in this kind of sentence will strongly depend on these nouns.</p>
<p>If you didn't yet you definitely should check out the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is all you need</a>-Paper as well as the <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT</a>-Paper (at least the abstract), they explain in detail how the mechanisms and the pretraining process work.</p>
<p>Another great source to get a better understanding of how it really works is <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">Illustrated Transformer</a>.</p>
How bert is a bidirectional?
<p>Bert encoder takes the input and goes for the multi-head attention model. But how do they maintain sequence? Since current words don't take sequence of previous words. Besides, why is it bidirectional? Does it maintain forward and backward sequence like LSTM?</p>