title: Does BERT and other language attention model only share cross-word information in the initial embedding stage?
body: <p>I study visual attention models but have recently been reading up on BERT and other language attention models to fill a serious gap in my knowledge.</p>
  <p>I am a bit confused by what I seem to be seeing in these model architectures. Given a sentence like &quot;the cat chased the dog&quot;. I would have expected cross information streams between the embeddings of each word. For example, I would have expected to see a point in the model where the embedding for &quot;cat&quot; is combined with the embedding for &quot;dog&quot;, in order to create the attention mask.</p>
  <p>Instead what I seem to be seeing, (correct me if I am wrong) is that the embedding of a word like &quot;cat&quot; is initially set up to include information about the words around them. So that each embedding of each word includes all of the other words around them. Then each of these embeddings are passed through the model in parallel. This seems weird to me and redundant. Why would they set up the model in this way?</p>
  <p>If we were to block out cat. &quot;the ... chased the dog.&quot; Would we then, during inference, only need to send the &quot;...&quot; embedding through the model?</p>