<p>According to the doc of <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LayerNormalization" rel="nofollow noreferrer"><code>tf.keras.layers.LayerNormalization</code></a> in <code>TF 2.4.1</code>, <a href="https://github.com/tensorflow/tensorflow/blob/v2.5.0/tensorflow/python/keras/layers/normalization.py#L1066-L1073" rel="nofollow noreferrer">source</a>:</p>
<blockquote>
<p>Note that other implementations of layer normalization may choose to
define <code>gamma</code> and <code>beta</code> over a separate set of axes from the axes
being normalized across. For example, Group Normalization (<a href="https://arxiv.org/abs/1803.08494" rel="nofollow noreferrer">Wu et al.
2018</a>) with a group size of 1 corresponds to a Layer Normalization
that normalizes across height, width, and channel and has <code>gamma</code> and
<code>beta</code> span only the channel dimension. <strong>So, this Layer Normalization
implementation will not match a Group Normalization layer with group
size set to 1.</strong></p>
</blockquote>
Why does tfa.layers.GroupNormalization(groups=1) produce different output than LayerNormalization?
<p>From the <strong>group normalization</strong> documentation in <code>tensorflow</code> addons, it states that the group norm layer should become layer normalization if the number of groups is set to one.</p>
<p>However when I try this by calling the layers one a test tensor the results differ.  It appears that the group norm layer computes the mean and variance across the time as well as the channel axis, whereas the layer norm computes it for each channel's vector independently.</p>
<p>Is this a bug or am I missing something?  The current behavior of layer norm is actually desirable for what I am doing.</p>
<p>Here is the documentation for <a href="https://www.tensorflow.org/addons/api_docs/python/tfa/layers/GroupNormalization" rel="nofollow noreferrer">GroupNormalization</a>:</p>
<pre><code>In [5]: x = tf.constant([[[1, 2], [3, 40]], [[1 , -1], [2, 200]]], dtype = tf.float32)
In [6]: tf.keras.layers.LayerNormalization()(x)                                                                                                               
Out[6]: 
&lt;tf.Tensor: shape=(2, 2, 2), dtype=float32, numpy=
array([[[-0.99800587,  0.99800587],
        [-0.99999857,  0.99999857]],

       [[ 0.9995002 , -0.9995002 ],
        [-1.        ,  1.        ]]], dtype=float32)&gt;

In [7]: tfa.layers.GroupNormalization(groups = 1)(x)                                                                                                          
Out[7]: 
&lt;tf.Tensor: shape=(2, 2, 2), dtype=float32, numpy=
array([[[-0.6375344 , -0.57681686],
        [-0.5160993 ,  1.7304504 ]],

       [[-0.5734435 , -0.5966129 ],
        [-0.5618587 ,  1.7319152 ]]], dtype=float32)&gt;
</code></pre>