title: What is weight decay loss?
body: <p>I have started recently with ML and TensorFlow. While going through the <a href="https://www.tensorflow.org/tutorials/deep_cnn" rel="nofollow noreferrer">CIFAR10-tutorial</a> on the website I came across a paragraph which is a bit confusing to me:</p>
  
  <blockquote>
    <p>The usual method for training a network to perform N-way classification is multinomial logistic regression, aka. softmax regression. Softmax regression applies a softmax nonlinearity to the output of the network and calculates the cross-entropy between the normalized predictions and a 1-hot encoding of the label. For regularization, we also apply the usual <strong>weight decay losses</strong> to all learned variables. The objective function for the model is the sum of the cross entropy loss and all these weight decay terms, as returned by the loss() function.</p>
  </blockquote>
  
  <p>I have read a few answers on what is weight decay on the forum and I can say that it is used for the purpose of regularization so that values of weights can be calculated to get the minimum losses and higher accuracy.</p>
  
  <p>Now in the text above I understand that the <code>loss()</code> is made of cross-entropy loss(which is the difference in prediction and correct label values) and weight decay loss. </p>
  
  <p>I am clear on cross entropy loss but what is this weight decay loss and why not just weight decay? How is this loss being calculated?</p>