According to the paper introducting Batch Normalization, the actual BN function is given as:

  • Input: Values of $x$ over a mini-batch $\mathcal B = \{x_{1,\ldots,m}\}$; parameters to be learned $\gamma,\beta$.
  • Output: $\{y_i = \mathrm{BN}_{\gamma,\beta}(x_i)\}$.

$\mu_{\mathcal B} \leftarrow \frac1m \sum_{i = 1}^m x_i$

$\sigma^2_{\mathcal B} \leftarrow \frac1m \sum_{i=1}^m (x_i - \mu_{\mathcal B})^2$

$\hat x_i \leftarrow \frac{x_i - \mu_{\mathcal B}}{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}$

$y_i \leftarrow \gamma \hat x_i + \beta \equiv \mathrm{BN}_{\gamma,\beta}(x_i)$

(Here, $\epsilon$ is some small constant added for numerical stability. The above is an almost exact copy of the box Algorithm 1, in section 3 of the paper linked above.)

Now, $\gamma,\beta$ are learned parameters, as far as I can tell on the level of each mini-batch. In particular, for a fixed mini-batch they can take any value. It seems to me that this makes shifting by the mean and scaling by the standard deviation pointless. The resulting output values are given by $$ y_i = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}}x_i + \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B}. $$ Hence, if we define $$ \gamma' = \frac{\gamma}{{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}}, $$ $$ \beta' = \beta - \frac{\gamma }{\sqrt{\sigma_{\mathcal B}^2 + \epsilon}}\mu_{\mathcal B}, $$ we might as just have defined and learned values for $\beta',\gamma'$ and then returned $y_i = \gamma'x_i + \beta'$.

I presume that I misunderstand -- can someone explain where I went wrong?

  • $\begingroup$ I had the same question and found the answer here. TL;DR the explicit mean and variance values are redundant but they make it easier to train. $\endgroup$ – Timmmm Jan 16 at 17:00

I'm by no means an expert on this topic, but here are my thoughts.

I think the main point of batch normalization is that training of each network layer is unaffected by changes in scale of the preceding layers. The author wrote

$$BN(Wu) = BN((aW)u)$$

for any scalar $a$. Thus, also backpropagated error is unaffected as well. So there is no scale explosion, even when learning rate is high.

If you simply apply

$$BN(x) = y_i = \gamma_ix_i + \beta_i, \text{ where } x = Wu$$

the above no longer holds.

Parameters $\gamma$ and $\beta$ are a must have, because otherwise the normalized outputs $\hat{x_i}$ would be mostly close to $0$, which would hinder the network ability to fully utilize nonlinear transformations (authors give an example of sigmoid function, which is close to identity transformation in $0$ proximity)

Finally, one can argue that the input $y = Wx + b$ to the BN layer, as a combination of many factors, may resemble Gaussian distribution (CLT). So by studentizing it we keep the inputs stable and, hopefully, following standard normal distribution to some degree.

  • $\begingroup$ Unless I misunderstand, while it is true that $\mathrm{BN}(Wu) = \mathrm{BN}((aW)u)$, the function actually depends on two more arguments, which are hidden here. A more explicit equation would be $\mathrm{BN}_{\beta,\gamma}(Wu) = \mathrm{BN}_{\beta,\gamma}((aW)u)$. And that is all nice and true, but if $\beta,\gamma$ themselves may vary depending on which input we are talking about, the equation does not really tell us anything. I presume this is where my misunderstanding arises, but: I do still misunderstand. $\endgroup$ – Mees de Vries Apr 5 '17 at 20:17
  • $\begingroup$ @MeesdeVries Why do you think it doesn't tell us anything, what does it matter that $\beta_i$ and $\gamma_i$ vary for each $i$. $\endgroup$ – Łukasz Grad Apr 5 '17 at 20:35
  • $\begingroup$ @MeesdeVries $\gamma$ and $\beta$ are learned parameters, just like any others in network - weights and biases. You may even throw away bias nodes because $\beta$ acts as bias, and they don't change from batch to batch (maybe this is were you're stuck) you simply update them every batch according to batch gradient $\endgroup$ – Łukasz Grad Apr 5 '17 at 20:42
  • $\begingroup$ Ah, I see it now. Earlier on the page, $\gamma,\beta$ are introduced with an index (as $\gamma^{(k)}, \beta^{(k)}$, leading me in my haste to erroneously read that the values varied of $\beta,\gamma$ varied with the input; but these are just the coordinates of the (single, trained) vectors $\gamma,\beta$. Thank you! $\endgroup$ – Mees de Vries Apr 6 '17 at 8:41

I sort of have/had the same reaction when I read the paper to be honest, and I'm still trying to understand it myself :-) But the good point is, that makes me interested in the answer to this question; the bad point is my own might or might not contribute partly to the answer.

What I think might be the goal is, a layer learns from the gradients coming backwards from the layer in the front. Ideally those gradients will be teaching the layer to produce useful abstract features, like spiral detectors and so on. However, it may be that somehow the layers in front of and behind this layer are basically pulling this layers outputs up and down globally, shifting them about, rather than pushing it to learn actually useful feature abstractions.

So, maybe what is happening is that the BN layer, which is after the layer, before the activations, handles learning to shift up/down via just two per-output parameters, gamma and beta, meanwhile the layer itself is protected somewhat from these shifts, can concentrate on learning feature abstractions. I'm not entirely convinced this is the 'right' explanation, but seems somewhat plausible?

  • $\begingroup$ Sorry, but I again feel that this answer does not address the question. It is not about general introduction of parameters $\gamma,\beta$ -- it is about having them specifically after a normalization, which seems (I presume it is a misunderstanding on my part) to undo the normalization, making that step pointless. $\endgroup$ – Mees de Vries Apr 5 '17 at 13:41
  • $\begingroup$ It does undo the normalization, I agree. I'm guessing that the goal is it means the layer itself doesnt need to laern them? so the layer can produce outputs in whatever range, it makes no difference, since they will be immediately normalized, and gamma and beta applied. gamma and beta should be relativley simple to learn, compared to shifting the weights of the entire layer? $\endgroup$ – Hugh Perkins Apr 5 '17 at 14:11

Correct me if I am wrong... The BN normalization values are calculated for each components to normalize the input to a neuron (i.e. elliptic distribution becomes circular). The resulting distribution may be too wide or too narrow and shifted to "activate" well the activation function. To my understanding, the gamma and beta learnt parameters are constants applied to all components (not gamma and beta values per component) learnt, re-scaling and shifting optimally the more circular distribution before applying the activation function. Thus not equivalent to only learn beta and gamma (which is already learnt by backprop in W and b coef) without prior normalization.


Your Answer

