title: Neural network: what is the order of parameter updates in this case?
body: <p>Input: <code>x, target</code></p>
  <p>Network:</p>
  <pre><code>P = torch.ones(x.shape)
  gamma = 1
  mask = W*x + b1 
  mask_x = mask*x
  y = mask_x*P
  y = W2*x + b2
  P = (gamma - mask_x)*P
  loss = CrossEntropy(y,target)
  </code></pre>
  <p>How does parameter <code>P</code> update in this case? I seem that in loss backward, <code>P</code> will update by gradient, however, here, I also update <code>P</code> by <code>P = (gamma - mask_x)*P</code>. I'm confused about the order of parameter's update.</p>
  <p>eg.
  Which is correct?</p>
  <pre><code>P = P + lr*grad_p
  mask_x = mask_x + lr*grad_m
  P = (gamma -mask_x)*P
  </code></pre>
  <p>Or</p>
  <pre><code>P = (gamma -mask_x)*P
  P = P + lr*grad_p
  mask_x = mask_x + lr*grad_m
  </code></pre>
  <p><code>grad_m</code> and <code>grad_p</code> are the gradient of parameter <code>mask_x</code> and <code>P</code> respectively.</p>