<p>I wanted to add more information to this question since there are some more recent works in this area. Your intuition</p>

<blockquote>
  <p>use instance normalisation for image classification where class label
  should not depend on the contrast of input image</p>
</blockquote>

<p>is partly correct. I would say that a pig in broad daylight is still a pig when the image is taken at night or at dawn. However, this does not mean using instance normalization across the network will give you better result. Here are some reasons:</p>

<ol>
<li>Color distribution still play a role. It is more likely to be a apple than an orange if  it has a lot of red.</li>
<li>At later layers, you can no longer imagine instance normalization acts as contrast normalization. Class specific details will emerge in deeper layers and normalizing them by instance will hurt the model's performance greatly.</li>
</ol>

<p><a href="https://arxiv.org/abs/1807.09441" rel="noreferrer">IBN-Net</a> uses both batch normalization and instance normalization in their model. They only put instance normalization in early layers and have achieved improvement in both accuracy and ability to generalize. They have open sourced code <a href="https://github.com/XingangPan/IBN-Net" rel="noreferrer">here</a>.</p>

<p><a href="https://i.stack.imgur.com/S74SW.png" rel="noreferrer"><img src="https://i.stack.imgur.com/S74SW.png" alt="enter image description here"></a></p>
Instance Normalisation vs Batch normalisation
<p>I understand that Batch Normalisation helps in faster training by turning the activation towards unit Gaussian distribution and thus tackling vanishing gradients problem. Batch norm acts is applied differently at training(use mean/var from each batch) and test time (use finalized running mean/var from training phase).</p>

<p>Instance normalisation, on the other hand, acts as contrast normalisation as mentioned in this paper <a href="https://arxiv.org/abs/1607.08022" rel="noreferrer">https://arxiv.org/abs/1607.08022</a> . The authors mention that the output stylised images should be not depend on the contrast of the input content image and hence Instance normalisation helps. </p>

<p>But then should we not also use instance normalisation for image classification where class label should not depend on the contrast of input image. I have not seen any paper using instance normalisation in-place of batch normalisation for classification. What is the reason for that? Also, can and should batch and instance normalisation be used together. I am eager to get an intuitive as well as theoretical understanding of when to use which normalisation. </p>