title: PyTorch Gradient Descent
body: <p>I am trying to manually implement gradient descent in PyTorch as a learning exercise. I have the following to create my synthetic dataset:</p>
  
  <pre><code>import torch
  torch.manual_seed(0)
  N = 100
  x = torch.rand(N,1)*5
  # Let the following command be the true function
  y = 2.3 + 5.1*x
  # Get some noisy observations
  y_obs = y + 2*torch.randn(N,1)
  </code></pre>
  
  <p>Then I create my predictive function (<code>y_pred</code>) as shown below.</p>
  
  <pre><code>w = torch.randn(1, requires_grad=True)
  b = torch.randn(1, requires_grad=True)
  y_pred = w*x+b
  mse = torch.mean((y_pred-y_obs)**2)
  </code></pre>
  
  <p>which uses MSE to infer the weights <code>w,b</code>. I use the block below to update the values according to the gradient.</p>
  
  <pre><code>gamma = 1e-2
  for i in range(100):
    w = w - gamma *w.grad
    b = b - gamma *b.grad
    mse.backward()
  </code></pre>
  
  <p>However, the loop only works in the first iteration. <strong>The second iteration onwards <code>w.grad</code> is set to <code>None</code>.</strong> I am fairly sure the reason this happens is because I am setting w as a function of it self (I might be wrong). </p>
  
  <p>The question is how do I update the weights properly with the gradient information?</p>