title: How to interpret the gradient norms in convolution neural networks?
body: <p>It might be basic thing, but I am unable to understand the interpretation of computing the gradient norms.</p>
  
  <p>For example, I am working on binary segmentation using architecture by stacking multiple U-Nets with each net has encoder similar to Resnet34. I trained this network and I obtain the gradient mean and norm values as below:</p>
  
  <ul>
  <li><blockquote>
    <p>conv1 has mean grad of -1.77767194275e-14</p>
  </blockquote></li>
  <li><blockquote>
    <p>conv1 has norm grad of 2.57230658463e-05</p>
  </blockquote></li>
  <li><blockquote>
    <p>layer1 has mean grad of 1.27381299952e-11</p>
  </blockquote></li>
  <li><blockquote>
    <p>layer1 has norm grad of 0.000390226632589</p>
  </blockquote></li>
  <li><blockquote>
    <p>layer2 has mean grad of -2.3107595698e-11</p>
  </blockquote></li>
  <li><blockquote>
    <p>layer2 has norm grad of 0.000265486567514</p>
  </blockquote></li>
  </ul>
  
  <p>All the values seems to be very low. I am not able to get shall I consider them as vanishing gradient problem or gradients can be really low depending upon the problem.</p>
  
  <p>If the gradients can be really low, what is best way to check for vanishing gradient problem?</p>
  
  <p>Any advice is appreciated to analyze the gradients data. Thanks in advance!</p>