title: Understanding the keras.layers.BatchNormalization computation
body: <p>I am playing with the <strong>BatchNormalization</strong> layer, and I can't quite figure out the numerical results I get.</p>
  
  <p>Let's consider we use BatchNormalization for computer vision.</p>
  
  <p>We have 4D tensors.</p>
  
  <p>Dimensions are: <strong>batch size</strong>, <strong>image height</strong>, <strong>image width</strong>, <strong>channels</strong>.</p>
  
  <p>If I understand correctly, what BatchNormalization will do is:</p>
  
  <ol>
  <li>At training time:
  
  <ul>
  <li>for each batch, compute the mean <strong>MU</strong> and the standard deviation <strong>SIGMA</strong>. This is done per channel, and accross all rows and all columns of all images of the batch.</li>
  <li>keep an exponential moving average of <strong>MU</strong> (say <strong>MÛ</strong>) and of <strong>SIGMA</strong> (say <strong>SIĜMA</strong>) accross all batches</li>
  <li>use <strong>MÛ</strong> and <strong>SIĜMA</strong> to normalize pixels: <strong><em>normalized_pixel = ((input_pixel - MÛ) / sqrt(SIĜMA))</em></strong></li>
  <li>an hyper-parameter <strong>epsilon</strong> is added to <strong>SIĜMA</strong> to prevent division by zero if <strong>SIĜMA</strong> becomes null at one point during training: <strong><em>normalized_pixel = ((input_pixel - MÛ) / sqrt(SIĜMA + epsilon))</em></strong></li>
  <li>use a scale parameter <strong>GAMMA</strong> and an offset parameter <strong>BETA</strong> to re-scale normalized pixel: <strong><em>output_pixel = ((GAMMA x normalized_pixel) + BETA)</em></strong></li>
  <li><strong>GAMMA</strong> and <strong>BETA</strong> are trainable parameters, they are optimized during training</li>
  </ul></li>
  <li>At inference time:
  
  <ul>
  <li><strong>MÛ</strong> and <strong>SIĜMA</strong> are now fixed parameters, just like <strong>GAMMA</strong> and <strong>BETA</strong></li>
  <li>Same computations apply</li>
  </ul></li>
  </ol>
  
  <p>Now, here comes my question...</p>
  
  <p>First, I am only interested in what happens at inference time.
  I don't care about training, and I consider <strong>MÛ</strong>, <strong>SIĜMA</strong>, <strong>GAMMA</strong> and <strong>BETA</strong> to be fixed parameters.</p>
  
  <p>I wrote a piece of python to test BatchNormalization on a <strong>(1, 3, 4, 1)</strong> tensor.
  Since there is only one channel, <strong>MÛ</strong>, <strong>SIĜMA</strong>, <strong>GAMMA</strong> and <strong>BETA</strong> have only 1 element each.</p>
  
  <p>I chose <strong>MÛ = 0.0</strong>, <strong>SIĜMA = 1.0</strong>, <strong>GAMMA = 1.0</strong> and <strong>BETA = 0.0</strong>, so that BatchNormalization has no effect.</p>
  
  <p>Here is the code:</p>
  
  <pre><code>#!/usr/bin/env python
  # -*- coding: utf-8 -*-
  
  import numpy
  import keras
  import math
  
  input_batch                         =   numpy.array(
                                              [[
                                                  [[ 1.0], [ 2.0], [ 3.0], [ 4.0]],
                                                  [[ 5.0], [ 6.0], [ 7.0], [ 8.0]],
                                                  [[ 9.0], [10.0], [11.0], [12.0]]
                                              ]],
                                              dtype=numpy.float32
                                          )
  
  
  MU                                  = 0.0
  SIGMA                               = 1.0
  GAMMA                               = 1.0
  BETA                                = 0.0
  
  input_layer                         =   keras.layers.Input(
                                              shape = (
                                                          None,
                                                          None,
                                                          1
                                                      )
                                          )
  
  BatchNormalization_layer            =   keras.layers.BatchNormalization(
                                              axis=-1,
                                              #epsilon=0.0,
                                              center=True,
                                              scale=True
                                          )(
                                              input_layer
                                          )
  
  model                               =   keras.models.Model(
                                              inputs  = [input_layer],
                                              outputs = [BatchNormalization_layer]
                                          )
  
  model.layers[1].set_weights(
      (
          numpy.array([GAMMA], dtype=numpy.float32),
          numpy.array([BETA],  dtype=numpy.float32),
          numpy.array([MU],    dtype=numpy.float32),
          numpy.array([SIGMA], dtype=numpy.float32),
      )
  )
  
  print model.predict(input_batch)
  
  print ((((input_batch - MU) / math.sqrt(SIGMA)) * GAMMA) + BETA)
  </code></pre>
  
  <p>When I write explicitely the computation <code>((((input_batch - MU) / math.sqrt(SIGMA)) * GAMMA) + BETA)</code> using numpy, I get the expected results.</p>
  
  <p>However, when I use the keras.layers.BatchNormalization layer to perform the computation, I get similar results, only there are some kind of rounding errors or imprecisions:</p>
  
  <pre><code>Using TensorFlow backend.
  [[[[ 0.9995004]
     [ 1.9990008]
     [ 2.9985013]
     [ 3.9980016]]
  
    [[ 4.997502 ]
     [ 5.9970026]
     [ 6.996503 ]
     [ 7.996003 ]]
  
    [[ 8.995503 ]
     [ 9.995004 ]
     [10.994504 ]
     [11.994005 ]]]]
  [[[[ 1.]
     [ 2.]
     [ 3.]
     [ 4.]]
  
    [[ 5.]
     [ 6.]
     [ 7.]
     [ 8.]]
  
    [[ 9.]
     [10.]
     [11.]
     [12.]]]]
  </code></pre>
  
  <p>When I play with the values of MU*, SIGMA*, GAMMA and BETA, the output is affected as expected, so I believe I provide the parameters correctly to the layer.</p>
  
  <p>I also tried to set the hyper-parameter epsilon of the layer to 0.0. It changes the results a little bit, but doe snot solve the issue.</p>
  
  <pre><code>Using TensorFlow backend.
  [[[[ 0.999995 ]
     [ 1.99999  ]
     [ 2.999985 ]
     [ 3.99998  ]]
  
    [[ 4.999975 ]
     [ 5.99997  ]
     [ 6.9999647]
     [ 7.99996  ]]
  
    [[ 8.999955 ]
     [ 9.99995  ]
     [10.999945 ]
     [11.99994  ]]]]
  [[[[ 1.]
     [ 2.]
     [ 3.]
     [ 4.]]
  
    [[ 5.]
     [ 6.]
     [ 7.]
     [ 8.]]
  
    [[ 9.]
     [10.]
     [11.]
     [12.]]]]
  </code></pre>
  
  <p>Can someone explain what is going on?</p>
  
  <p>Thanks,</p>
  
  <p>Julien</p>