<p>I have finally figured out the problem.</p>

<p>Batch normalization learns two parameters during training and uses them for inference. Thus it is necessary to change its behaviour using <code>eval()</code> to tell not to modify them any further.</p>

<p>I then scrutinizingly checked the <a href="https://arxiv.org/pdf/1602.07868.pdf" rel="noreferrer">weight normalization</a> paper and found it to be 'inherently deterministic'. It simply decouples the original weight vectors as product of two quantities as shown below.</p>

<pre><code>w = g . v
</code></pre>

<p>Obviously either you use LHS for computing output or RHS it does not matter. However by decoupling it into two vectors and passing them to optimizer and deleting the <code>w</code> parameter better training is achieved. For reasons refer the paper where things are nicely described.</p>

<p>Thus it does not matter if weight normalization is removed or not during testing. To validate this I tried the following small code.</p>

<pre><code>import torch
import torch.nn as nn
from torch.nn.utils import weight_norm as wn
from torch.nn.utils import remove_weight_norm as wnr

# define the model 'm'
m = wn(nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=1, bias=True))

ip = torch.rand(1,1,5,5)
target = torch.rand(1,1,5,5)
l1 = torch.nn.L1Loss()
optimizer = torch.optim.Adam(m.parameters())



# begin training
for _ in range(5):
    out = m(ip)
    loss = l1(out,target)
    loss.backward()
    optimizer.step()

with torch.no_grad():
    m.eval()
    print('\no/p after training with wn: {}'.format(m(ip)))
    wnr(m)
    print('\no/p after training without wn: {}'.format(m(ip)))

# begin testing
m2 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3,padding=1, bias=True)
m2.load_state_dict(m.state_dict())

with torch.no_grad():
    m2.eval()
    out = m2(ip)
    print('\nOutput during testing and without weight_norm: {}'.format(out))
</code></pre>

<p>And the output is below,</p>

<pre><code>o/p after training with wn: 
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])

o/p after training without wn: 
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])

Output during testing and without weight_norm: 
tensor([[[[0.0509, 0.3286, 0.4612, 0.1795, 0.0307],
          [0.1846, 0.3931, 0.5713, 0.2909, 0.4026],
          [0.1716, 0.5971, 0.4297, 0.0845, 0.6172],
          [0.2938, 0.2389, 0.4478, 0.5828, 0.6276],
          [0.1423, 0.2065, 0.5024, 0.3979, 0.3127]]]])
</code></pre>

<p>Please see that all the values are exactly same as only reparameterization is happening.</p>

<p>Regarding,</p>

<blockquote>
  <p>Then I tested two models using C++ code with libtorch. But the results are not the same.</p>
</blockquote>

<p>See <a href="https://github.com/pytorch/pytorch/issues/21275" rel="noreferrer">https://github.com/pytorch/pytorch/issues/21275</a> which reports a bug with TorchScript.</p>

<p>And regarding,</p>

<blockquote>
  <p>I am wondering what does weight_norm do in inference? Is it usefull?</p>
</blockquote>

<p>The answer is it does nothing. you do <code>x * 2</code> or <code>x * (1+1)</code> does not matter. It is not useful but not harmful either. So better remove it.</p>
Weight Normalization in PyTorch
<p>An important weight normalization technique was introduced in <a href="https://arxiv.org/pdf/1602.07868.pdf" rel="nofollow noreferrer">this paper</a> and has been included in PyTorch since long as follows:</p>

<pre><code>    from torch.nn.utils import weight_norm
    weight_norm(nn.Conv2d(in_channles, out_channels))
</code></pre>

<p>From the <a href="https://pytorch.org/docs/1.3.1/nn.html#torch.nn.utils.weight_norm" rel="nofollow noreferrer">docs</a> I get to know, <code>weight_norm</code> does re-parametrization before each <code>forward()</code> pass. But I am not sure if this re-parameterization is also happening during the inference when everything is running inside <code>with torch.no_grad()</code> and the model is set to <code>eval()</code> mode.</p>

<p>Can someone please help me know if <code>weight_norm</code> is active only during training or during the inference mode as described above?</p>

<p>Thank you</p>