title: Unknown behavior of hooks on batch norm in pytorch
body: <p>I try to freeze the batch_norm layer and analyse their inputs/outputs with forward hooks</p>
  <p>For fixed BN layers, I just couldn't understand why the hooked output is different from the output reproduced by the hooked input.</p>
  <p>Really appreciate that if anyone could help me</p>
  <p>Here's the code:</p>
  <pre><code>import torch
  import torchvision
  import numpy
  
  
  def set_bn_eval(m):
      classname = m.__class__.__name__
      if classname.find('BatchNorm') != -1:
          m.eval()
  
  image = torch.randn((1, 3, 224, 224))
  res = torchvision.models.resnet50(pretrained=True)
  res.apply(set_bn_eval)
  b = res(image)
  layer_out = []
  layer_in = []
  
  def layer_hook(mod, inp, out):
      layer_out.append(out)
      layer_in.append(inp[0])
  
  for name, key in res.named_modules():
      hook = key.register_forward_hook(layer_hook)
      res(image)
      hook.remove()
      out = layer_out.pop()
      inp = layer_in.pop()
      try:
          assert (out.equal(key(inp)))
      except AssertionError:
          print(name)
          break
  </code></pre>