title: How to decide which mode to use for 'kaiming_normal' initialization
body: <p>I have read several codes that do layer initialization using <code>nn.init.kaiming_normal_()</code> of PyTorch. Some codes use the <code>fan in</code> mode which is the default. Of the many examples, one can be found <a href="https://github.com/Xiaoming-Yu/RAGAN/blob/master/models/model.py#L50" rel="noreferrer">here</a> and shown below.</p>
  
  <pre><code>init.kaiming_normal(m.weight.data, a=0, mode='fan_in')
  </code></pre>
  
  <p>However, sometimes I see people using the <code>fan out</code> mode as seen <a href="https://github.com/liminn/ICNet-pytorch/blob/master/models/base_models/resnetv1b.py#L126" rel="noreferrer">here</a> and shown below.</p>
  
  <pre><code>if isinstance(m, nn.Conv2d):
                  nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
  
  </code></pre>
  
  <p>Can someone give me some guidelines or tips to help me decide which mode to select? Further I am working on image super resolutions and denoising tasks using PyTorch and which mode will be more beneficial.</p>