<p><img src="https://jalammar.github.io/images/t/self-attention-matrix-calculation-2.png" alt="" /> <em>Taken from <a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">https://jalammar.github.io/illustrated-transformer/</a></em></p>
<p>Changing the order of words, will permute the order of the rows of <code>V</code>, but will also permute the order of the columns of the correlation matrix <code>Q x transpose(K)</code>. Thus the resulting output will be unchanged and positional information will be lost after the first self attention layer.</p>
<p>To solve this you encode the position into the embedding of each word, so the neural net can learn to take two embeddings and know how far they are apart no matter the order they are fed in.</p>
<p>From the abstract that claimed <a href="https://arxiv.org/abs/1905.04226" rel="nofollow noreferrer">positional encoding is not necessary</a>:</p>
<blockquote>
<p>The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering.</p>
</blockquote>
Is positional encoding necessary for transformer in language modeling?
<p>I am developing a language model like <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noreferrer">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a>.</p>

<p>It is not clear for me - whether positional encoding is neccessary here ?
As far as I understand - it is necessary for language translation task because the decoder should be able to position  the word from the previous output within the sequence from encoder.
But is it necessary in language modeling without the decoder ?</p>

<p>Is it possible that the words in the encoder output are shuffled ?</p>

<p><strong>Edit:</strong> </p>

<p>there are no explanations in the original paper. And I didn't find explanations in tutorials (like here <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/" rel="nofollow noreferrer">https://kazemnejad.com/blog/transformer_architecture_positional_encoding/</a>).</p>

<p>I don't understand this: </p>

<p>"As each word in a sentence simultaneously flows through the Transformer’s encoder/decoder stack, The model itself doesn’t have any sense of position/order for each word."</p>

<p>From my point of view - transformer encoder has info about the order because its input is an ordered sequence (similar to RNN).</p>

<p>I tried to remove positional encoding from the model. It works, but with a worse performance.</p>

<p>Is it useful to add such positional encoding to RNN ? Could it improve its performance ?</p>