<p>You should probably read an explanation about Batch Normalization, such as <a href="https://gab41.lab41.org/batch-normalization-what-the-hey-d480039a9e3b" rel="noreferrer">this one</a>. You can also take a look at <a href="https://www.tensorflow.org/api_docs/python/tf/nn/batch_normalization" rel="noreferrer">tensorflow's related doc</a>. </p>

<p>Basically, there are 2 ways you can do batch_norm, and both have problems dealing with batch size of 1:</p>

<ul>
<li><p>using a moving mean and variance pixel per pixel, so they are tensors of the same shape as each sample in your batch. This is the one used in @layog's answer, and (I think) in <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="noreferrer">the original paper</a>, and the most used.</p></li>
<li><p>Using a moving mean and variance over the entire image / feature space, so they are just vectors (rank 1) of shape <code>(n_channels,)</code>.</p></li>
</ul>

<p>In both cases, you'll have:</p>

<pre><code>output = gamma * (input - mean) / sigma + beta
</code></pre>

<p>Beta is often set to 0 and gamma to 1, since you have linear functions right after BN.</p>

<p><strong>During training</strong>, <code>mean</code> and <code>variance</code> are computed <em>accross the current batch</em>, which causes problem when it is of size 1:</p>

<ul>
<li>in the 1st case, you'll get <code>mean=input</code>, so <code>output=0</code></li>
<li>in the 2nd case, <code>mean</code> will be the average value over all pixels, so it's better; but if your width and height are also 1, then you get <code>mean=input</code> again, so you get <code>output=0</code>.</li>
</ul>

<p>I think most people (and the original method) use the 1st way, which is why you'll get 0 (although TF doc seems to suggest that the 2nd method is usual too). The argument in the link you're providing seems to be considering the 2nd method.</p>

<p>In any case (whichever you're using), with BN you'll only get good results if you use a bigger batch size (say, at least 10).</p>
Tensorflow and Batch Normalization with Batch Size==1 => Outputs all zeros
<p>I have a question about the understanding of the BatchNorm (BN later on).</p>

<p>I have a convnet working nicely, I was writing tests to check for shape and outputs range. And I noticed that when I set the batch_size = 1, my model outputs zeros (logits and activations).</p>

<p>I prototyped the simplest convnet with BN:</p>

<p><strong>Input => Conv + ReLU => BN => Conv + ReLU => BN => Conv Layer + Tanh</strong></p>

<p>The model is initialized with <em>xavier initialization</em>. I guess that BN <strong>during training</strong> do some calculations that require Batch_size > 1.</p>

<p>I have found an issue in PyTorch that seems to talk about this: <a href="https://github.com/pytorch/pytorch/issues/1381" rel="noreferrer">https://github.com/pytorch/pytorch/issues/1381</a></p>

<p>Could anyone explain this ? It's still a little blurry for me.</p>

<hr>

<p><strong>Example Run:</strong></p>

<p><strong>Important:</strong> Tensorlayer Library is required for this script to run: <em>pip install tensorlayer</em></p>

<pre><code>import tensorflow as tf
import tensorlayer as tl

import numpy as np

def conv_net(inputs, is_training):

    xavier_initilizer = tf.contrib.layers.xavier_initializer(uniform=True)
    normal_initializer = tf.random_normal_initializer(mean=1., stddev=0.02)

    # Input Layers
    network = tl.layers.InputLayer(inputs, name='input')

    fx = [64, 128, 256, 256, 256]

    for i, n_out_channel in enumerate(fx):

        with tf.variable_scope('h' + str(i + 1)):

            network = tl.layers.Conv2d(
                network,
                n_filter    = n_out_channel,
                filter_size = (5, 5),
                strides     = (2, 2),
                padding     = 'VALID',
                act         = tf.identity,
                W_init      = xavier_initilizer,
                name        = 'conv2d'
            )

            network = tl.layers.BatchNormLayer(
                network,
                act        = tf.identity,
                is_train   = is_training,
                gamma_init = normal_initializer,
                name       = 'batch_norm'
            )

            network = tl.layers.PReluLayer(
                layer  = network,
                a_init = tf.constant_initializer(0.2),
                name   ='activation'
            )

    ############# OUTPUT LAYER ###############

    with tf.variable_scope('h' + str(len(fx) + 1)):
        '''

        network = tl.layers.FlattenLayer(network, name='flatten')

        network = tl.layers.DenseLayer(
            network,
            n_units = 100,
            act     = tf.identity,
            W_init  = xavier_initilizer,
            name    = 'dense'
        )

        '''

        output_filter_size = tuple([int(i) for i in network.outputs.get_shape()[1:3]])

        network = tl.layers.Conv2d(
            network,
            n_filter    = 100,
            filter_size = output_filter_size,
            strides     = (1, 1),
            padding     = 'VALID',
            act         = tf.identity,
            W_init      = xavier_initilizer,

            name        = 'conv2d'
        )

        network = tl.layers.BatchNormLayer(
            network,
            act        = tf.identity,
            is_train   = is_training,
            gamma_init = normal_initializer,
            name       = 'batch_norm'
        )

        net_logits = network.outputs

        network.outputs = tf.nn.tanh(
            x        = network.outputs,
            name     = 'activation'
        )

        net_output = network.outputs

    return network, net_output, net_logits


if __name__ == '__main__':

    tf.logging.set_verbosity(tf.logging.DEBUG)

    #################################################
    #                MODEL DEFINITION               #
    #################################################

    PLH_SHAPE = [None, 256, 256, 3]

    input_plh = tf.placeholder(tf.float32, PLH_SHAPE, name='input_placeholder')

    convnet, net_out, net_logits = conv_net(input_plh, is_training=True)


    with tf.Session() as sess:
        tl.layers.initialize_global_variables(sess)

        convnet.print_params(details=True)

        #################################################
        #                  LAUNCH A RUN                 #
        #################################################

        for BATCH_SIZE in [1, 2]:

            INPUT_SHAPE = [BATCH_SIZE, 256, 256, 3]

            batch_data = np.random.random(size=INPUT_SHAPE)

            output, logits = sess.run(
                [net_out, net_logits],
                feed_dict={
                    input_plh: batch_data
                }
            )

            if tf.logging.get_verbosity() == tf.logging.DEBUG:
                print("\n\n###########################")

                print("\nBATCH SIZE = %d\n" % BATCH_SIZE)

            tf.logging.debug("output =&gt; Shape: %s - Mean: %e - Std: %f - Min: %f - Max: %f" % (
                output.shape,
                output.mean(),
                output.std(),
                output.min(),
                output.max()
            ))

            tf.logging.debug("logits =&gt; Shape: %s - Mean: %e - Std: %f - Min: %f - Max: %f" % (
                logits.shape,
                logits.mean(),
                logits.std(),
                logits.min(),
                logits.max()
            ))

            if tf.logging.get_verbosity() == tf.logging.DEBUG:
                print("###########################")
</code></pre>

<p><strong>Gives the following output:</strong></p>

<pre><code>###########################

BATCH SIZE = 1

DEBUG:tensorflow:output =&gt; Shape: (1, 1, 1, 100) - Mean: 0.000000e+00 - Std: 0.000000 - Min: 0.000000 - Max: 0.000000
DEBUG:tensorflow:logits =&gt; Shape: (1, 1, 1, 100) - Mean: 0.000000e+00 - Std: 0.000000 - Min: 0.000000 - Max: 0.000000
###########################


###########################

BATCH SIZE = 2

DEBUG:tensorflow:output =&gt; Shape: (2, 1, 1, 100) - Mean: -1.430511e-08 - Std: 0.760749 - Min: -0.779634 - Max: 0.779634
DEBUG:tensorflow:logits =&gt; Shape: (2, 1, 1, 100) - Mean: -4.768372e-08 - Std: 0.998715 - Min: -1.044437 - Max: 1.044437
###########################
</code></pre>