title: How to implement weight decay in tensorflow as in Caffe
body: <p>In Caffe we have a decay_ratio which is usually set as 0.0005. Then all trainable parameters, e.g., W matrix in FC6 will be decayed by:
  W = W * (1 - 0.0005)
  after we applied the gradient to it.</p>
  
  <p>I go through many tutorial tensorflow codes, but do not see how people implement this weight decay to prevent numerical problems (very large absolute values)</p>
  
  <p>I my experiences, I often run into numerical problems aften 100k iterations during training.</p>
  
  <p>I also go through related questions at stackoverflow, e.g., 
  <a href="https://stackoverflow.com/questions/34986911/how-to-set-weight-cost-strength-in-tensorflow">How to set weight cost strength in TensorFlow?</a>
  However, the solution seems a little different as implemented in Caffe.</p>
  
  <p>Does anyone has similar concerns? Thank you.</p>