<p>As per my understanding (as there is no clear explanation given anywhere) -</p>
<p>These are just multiple ways to implement the same concept.</p>
<p>Basic concept is - to use sine and cosine positional encodings to encode the embeddings as per their positions in the sentence. So, if the word '<strong>it</strong>' is present in 2 separate sentences, their embeddings will be different for both the sentences due to added positional encodings.</p>
<p>The idea is to have some way to implement the time-steps from RNNs in which positions of words in the sentence is encoded by default, due to the way RNNs are implemented.</p>
<p>So, if you notice the difference between equations - written in <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">paper</a> and the one implemented in the <a href="https://www.tensorflow.org/tutorials/text/transformer?hl=en#positional_encoding" rel="nofollow noreferrer">official tensorflow website</a> and the one implemented in the <a href="https://github.com/tensorflow/tensor2tensor/tree/5f9dd2db6d7797162e53adf152310ed13e9fc711/tensor2tensor/layers" rel="nofollow noreferrer">official tensor2tensor github repository</a>, they all are different.</p>
<p>In <strong>tensorflow website</strong>, they have merged sine and cosine encodings at odd and even positions -</p>
<pre><code>def get_angles(pos, i, d_model):
  angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
  return pos * angle_rates

def positional_encoding(position, d_model):
  angle_rads = get_angles(np.arange(position)[:, np.newaxis],
                          np.arange(d_model)[np.newaxis, :],
                          d_model)

  # apply sin to even indices in the array; 2i
  angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])

  # apply cos to odd indices in the array; 2i+1
  angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])

  pos_encoding = angle_rads[np.newaxis, ...]

  return tf.cast(pos_encoding, dtype=tf.float32)
</code></pre>
<p>whereas in the <strong>tensor2tensor implementation</strong>, they have concatenated sine and cosine encodings. (Channels here mean d_model, and signal means position encodings) (Notice the divisor in the implementation. Its <em>d_model-1</em> rather than <em>d_model</em>)</p>
<pre><code>  channels = common_layers.shape_list(x)[2]
  num_timescales = channels // 2
  log_timescale_increment = (
      math.log(float(max_timescale) / float(min_timescale)) /
      (tf.to_float(num_timescales) - 1))
  inv_timescales = min_timescale * tf.exp(
      tf.to_float(tf.range(num_timescales)) * -log_timescale_increment)
  scaled_time = (
      tf.expand_dims(tf.to_float(position), 2) * tf.expand_dims(
          tf.expand_dims(inv_timescales, 0), 0))
  signal = tf.concat([tf.sin(scaled_time), tf.cos(scaled_time)], axis=2)
  signal = tf.pad(signal, [[0, 0], [0, 0], [0, tf.mod(channels, 2)]])
  signal = common_layers.cast_like(signal, x)
  return x + signal
</code></pre>
<p>Ultimately, what matters is to have different encodings for each word in the sequence which, with sine and cosine functions applied to different frequency and phase, gives different output. Just point is to remember to continue using same equation across the model. Once you change the way of implementation, you lose the correct positional encodings.</p>
Explanation about i//2 in positional encoding in tensorflow tutorial about transformers
<p>I was implementing the transformer architecture in tensorflow.</p>
<p>I was following the tutorial : <a href="https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline</a></p>
<p>They implement the positional encoding in this way:</p>
<pre><code>angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
</code></pre>
<p>However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?</p>
<p><a href="https://i.stack.imgur.com/67ADh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/67ADh.png" alt="pos" /></a></p>
<p>Thankyou</p>