<p>According to <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_" rel="noreferrer">documentation</a>:</p>

<blockquote>
  <p>Choosing 'fan_in' preserves the magnitude of the variance of the
  weights in the forward pass. Choosing 'fan_out' preserves the
  magnitudes in the backwards pass.</p>
</blockquote>

<p>and according to <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="noreferrer">Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015)</a>: </p>

<blockquote>
  <p>We  note  that  it  is  sufficient  to  use  either  Eqn.(14)  or
  Eqn.(10)</p>
</blockquote>

<p>where Eqn.(10) and Eqn.(14) are <code>fan_in</code> and <code>fan_out</code> appropriately. Furthermore:</p>

<blockquote>
  <p>This means that if the initialization properly scales the backward
  signal, then this is also the case for the forward signal; and vice
  versa. For all models in this paper, both forms can make them converge</p>
</blockquote>

<p><strong>so all in all it doesn't matter much</strong> but it's more about what you are after. I assume that if you suspect your backward pass might be more "chaotic" (greater variance) it is worth changing the mode to <code>fan_out</code>. This might happen when the loss oscillates a lot (e.g. very easy examples followed by very hard ones).</p>

<p>Correct choice of <code>nonlinearity</code> is more important, where <code>nonlinearity</code> is the activation you are using <strong>after</strong> the layer you are initializaing currently. Current defaults set it to <code>leaky_relu</code> with <code>a=0</code>, which is effectively the same as <code>relu</code>. If you are using <code>leaky_relu</code> you should change <code>a</code> to it's slope.</p>
How to decide which mode to use for 'kaiming_normal' initialization
<p>I have read several codes that do layer initialization using <code>nn.init.kaiming_normal_()</code> of PyTorch. Some codes use the <code>fan in</code> mode which is the default. Of the many examples, one can be found <a href="https://github.com/Xiaoming-Yu/RAGAN/blob/master/models/model.py#L50" rel="noreferrer">here</a> and shown below.</p>

<pre><code>init.kaiming_normal(m.weight.data, a=0, mode='fan_in')
</code></pre>

<p>However, sometimes I see people using the <code>fan out</code> mode as seen <a href="https://github.com/liminn/ICNet-pytorch/blob/master/models/base_models/resnetv1b.py#L126" rel="noreferrer">here</a> and shown below.</p>

<pre><code>if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')

</code></pre>

<p>Can someone give me some guidelines or tips to help me decide which mode to select? Further I am working on image super resolutions and denoising tasks using PyTorch and which mode will be more beneficial.</p>