<p>This is only a toy model with no neurons. There is no optimization going on here. Batch normalization won't change your <code>X</code> variable because by definition it is a constant.</p>

<p>What it does is: <em>in the process of training a neural network</em>, it <em>transforms</em> your outputs from some layer into normalized inputs to the next layer such that it helps to train the next layer's weights. I am not a kerns user, but I'd guess you might be able to check the normalized outputs of some layer only by inspecting the tensorflow nodes directly (if then)</p>

<p>To answer the title of your question, Batch Normalization in itself is just <a href="https://en.wikipedia.org/wiki/Standard_score#Calculation" rel="nofollow noreferrer">standard z-score normalization</a>. 
It is the same as subtracting the mean and dividing by the standard deviation of the series. </p>

<p>In mathematical notation,</p>

<p><a href="https://i.stack.imgur.com/NbpBu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NbpBu.png" alt="enter image description here"></a></p>

<p>In code, where <code>arr</code> is a numpy array,</p>

<pre><code>(arr - arr.mean(axis=0))/arr.std(axis=0, ddof=1)
</code></pre>

<p>The idea of normalizing is to get your distribution closer to a standard normal with mean 0 and standard deviation 1 i.e. ~ N(0,1).</p>

<p>It has been discussed lately (e.g. <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">here</a> and <a href="https://papers.nips.cc/paper/7996-understanding-batch-normalization.pdf" rel="nofollow noreferrer">here</a>) that by renormalizing your batches you can train your Neural Networks faster by reducing internal covariate shift.</p>
How keras.layers.Batchnomalization works?
<p>I tried batch normalization for toy set [[1,2],[5,4]. 
Normalizaing among axis=0, we get </p>

<pre><code>#[[-1/sqrt(2),-1/sqrt(2)],[1/sqrt(2), 1/sqrt(2)]]
</code></pre>

<p>However, my layer(axis=0) and layer(axis=1) both give incorrect result.</p>

<pre><code>X = tf.constant([[1,2],[5,4]],dtype = tf.float32)
layer = keras.layers.BatchNormalization()
hidden = layer(X)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer(axis=0))
    print(sess.run(layer.trainable_weights))
    print(sess.run(hidden))

#results
#[array([1., 1.], dtype=float32), array([0., 0.], dtype=float32)]
#[[0.9995004 4.997502 ]
# [1.9990008 3.9980016]]


X = tf.constant([[1,2],[5,4]],dtype = tf.float32)
layer = keras.layers.BatchNormalization()
hidden = layer(X)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer(axis=1))
    print(sess.run(layer.trainable_weights))
    print(sess.run(hidden))

#results
#[array([1., 1.], dtype=float32), array([0., 0.], dtype=float32)]
#[[0.9995004 4.997502 ]
# [1.9990008 3.9980016]]
</code></pre>

<p>gamma=1 and beta=0 as trainable_weights shows. Then how this layer works? </p>