title: Weak optimizers in Pytorch
body: <p>Consider a simple line fitting <code>a * x + b = x</code>, where <code>a</code>, <code>b</code> are the optimized parameters and <code>x</code> is the observed vector given by</p>
  
  <pre><code>import torch
  X = torch.randn(1000,1,1)
  </code></pre>
  
  <p>One can immediately see that the exact solution is <code>a=1</code>, <code>b=0</code> for any <code>x</code> and it can be found as easily as:</p>
  
  <pre><code>import numpy as np
  np.polyfit(X.numpy().flatten(), X.numpy().flatten(), 1)
  </code></pre>
  
  <p>I am trying now to find this solution by means of gradient descent in PyTorch, where the mean square error is used as an optimization criterion. </p>
  
  <pre><code>import matplotlib.pyplot as plt
  import numpy as np
  
  import torch
  import torch.nn as nn
  from torch.optim import Adam, SGD, Adagrad, ASGD 
  
  X = torch.randn(1000,1,1) # Sample data
  
  class SimpleNet(nn.Module): # Trivial neural network containing two weights
      def __init__(self):
          super(SimpleNet, self).__init__()
          self.f1 = nn.Linear(1,1)
  
      def forward(self, x):
          x = self.f1(x)
          return x
  
  # Testing default setting of 3 basic optimizers
  
  K = 500
  net = SimpleNet() 
  optimizer = Adam(params=net.parameters())
  Adam_losses = []
  optimizer.zero_grad()   # zero the gradient buffers
  for k in range(K):
      for b in range(1): # single batch
          loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
          loss.backward()
          optimizer.step()
          Adam_losses.append(float(loss.detach()))
  
  net = SimpleNet()
  optimizer = SGD(params=net.parameters(), lr=0.0001)
  SGD_losses = []
  optimizer.zero_grad()   # zero the gradient buffers
  for k in range(K):
      for b in range(1): # single batch
          loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
          loss.backward()
          optimizer.step()
          SGD_losses.append(float(loss.detach()))
  
  net = SimpleNet()     
  optimizer = Adagrad(params=net.parameters())
  Adagrad_losses = []
  optimizer.zero_grad()   # zero the gradient buffers
  for k in range(K):
      for b in range(1): # single batch
          loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
          loss.backward()
          optimizer.step()
          Adagrad_losses.append(float(loss.detach()))
  </code></pre>
  
  <p>The training progress in terms of loss evolution can be shown as 
  <a href="https://i.stack.imgur.com/rn9W8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rn9W8.png" alt="Convergence process of 3 optimizer algorithms"></a></p>
  
  <p>What is surprising for me is a very slow convergence of the algorithms in default setting. I have thus 2 questions:</p>
  
  <p>1) Is it possible to achieve an arbitrary small error (loss) purely by means of some Pytorch optimizer? Since the loss function is convex, it should be definitely possible, however, I am not able to figure out, how to achieve this using PyTorch. Note that the above 3 optimizers cannot do that - see the loss progress in log scale for 20000 iterations:
  <a href="https://i.stack.imgur.com/dYMFP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dYMFP.png" alt="Log-scale plot for 20000 training iterations"></a></p>
  
  <p>2) I am wondering how the optimizers can work well in complex examples, when they does not work well even in this extremely simple example. Or (and that is the second question) is it something wrong in their application above that I missed?</p>