title: What's the mistake in my Python version of a toy SVM?
body: <p>I'm currently going through Andrej Karpathy's <a href="http://karpathy.github.io/neuralnets/" rel="nofollow">Hacker's guide to Neural Networks</a>. In <em>Chapter 2: Machine Learning, Binary Classification</em>, he gives an example of a (very basic) SVM. Here's Karpathy's code:</p>
  
  <pre><code>var a = 1, b = -2, c = -1; // initial parameters
  for(var iter = 0; iter &lt; 400; iter++) {
    // pick a random data point
    var i = Math.floor(Math.random() * data.length);
    var x = data[i][0];
    var y = data[i][1];
    var label = labels[i];
  
  // compute pull
    var score = a*x + b*y + c;
    var pull = 0.0;
    if(label === 1 &amp;&amp; score &lt; 1) pull = 1;
    if(label === -1 &amp;&amp; score &gt; -1) pull = -1;
  
  // compute gradient and update parameters
    var step_size = 0.01;
    a += step_size * (x * pull - a); // -a is from the regularization
    b += step_size * (y * pull - b); // -b is from the regularization
    c += step_size * (1 * pull);
  }
  </code></pre>
  
  <p>And the following is my version, in Python:</p>
  
  <pre><code>import numpy
  import random
  
  X = numpy.array([[1.2, 0.7],
                   [-0.3, 0.5],
                   [-3, -1],
                   [0.1, 1.0],
                   [3.0, 1.1],
                   [2.1, -3]])
  
  labels = [1, -1, 1, -1, -1, 1]
  
  a = 1
  b = -2
  c = -1
  
  l = len(X)-1
  
  steps = 400
  
  for n in range(0, steps):
      i = random.randint(0, l)
      x = X[i][0]
      y = X[i][1]
      label = labels[i]
  
      if n == 0:
              for j in range(0, l+1):
                  x = X[j][0]
                  y = X[j][1]
                  label = labels[j]
                  score = a*x + b*y + c
                  print x,",",y,"--&gt;", label, "vs.", score
  
      score = a*x + b*y + c
      pull = 0.0
      if label == 1 and score &lt; 1:
          pull = 1
      if label == -1 and score &gt; -1:
          pull = -1
  
      step_size = 0.01
      a += step_size * (x * pull - a)
      b += step_size * (y * pull - b)
      c += step_size * (1 * pull)
  
      if n == steps-1:
          print ""
          for j in range(0, l+1):
              x = X[j][0]
              y = X[j][1]
              label = labels[j]
              score = a*x + b*y + c
              print x,",",y,"--&gt;", label, "vs.", score
  </code></pre>
  
  <p>The problem is, that even after more than the suggested 400 iterations, for some of the vectors, the parameters don't yield the correct label. </p>
  
  <p>Here's the output after 400 iterations:</p>
  
  <pre><code>1.2 , 0.7 --&gt; 1 vs. -0.939483353298
  -0.3 , 0.5 --&gt; -1 vs. -0.589208602761
  -3.0 , -1.0 --&gt; 1 vs. 0.651953448705
  0.1 , 1.0 --&gt; -1 vs. -0.921882586141
  3.0 , 1.1 --&gt; -1 vs. -1.44552077331
  2.1 , -3.0 --&gt; 1 vs. 0.896623596303
  </code></pre>
  
  <p>The first value after the "-->" is the correct label, second value is the score, i.e. learned label. </p>
  
  <p>All vector/learned labels are correct (in the sense of being assigned a value with the correct sign), except for the first one.</p>
  
  <p>I'm not sure what the reason is for this: Did I make a mistake in my code? I checked it a few times, but didn't find anything. Or am I forgetting something Python specific here. Or, finally, is there some ML related reason why the correct label isn't learned in this case. Doubt it though, otherwise it doesn't make sense that Karpathy got the right results.</p>
  
  <p>Any comments or help in figuring it out much appreciated.</p>