title: Python Minibatch Dictionary Learning
body: <p>I'd like to implement error tracking with dictionary learning in python, using sklearn's <a href="http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.MiniBatchDictionaryLearning.html" rel="nofollow">MiniBatchDictionaryLearning</a>, so that I can record how the error decreases over the iterations. I have two methods to do it, neither of which really worked. Set up:</p>
  
  <ul>
  <li><strong>Input data X</strong>, numpy array shape (n_samples, n_features) = (298143, 300). These are patches of shape (10, 10), generated from an image of shape (642, 480, 3).</li>
  <li><strong>Dictionary learning parameters</strong>: No. of columns (or atoms) = 100, alpha = 2, transform algorithm = OMP, total no. of iterations = 500 (keep it small first, just as a test case)</li>
  <li><p><strong>Calculating error</strong>: After learning the dictionary, I encode the original image again based on the learnt dictionary. Since both the encoding and the original are numpy arrays of the same shape (642, 480, 3), I'm just doing elementwise Euclidean distance for now: </p>
  
  <p>err = np.sqrt(np.sum(reconstruction - original)**2))</p></li>
  </ul>
  
  <p>I did a test run with these parameters, and the full fit was able to produce a pretty good reconstruction with a low error, so that's good.Now on to the two methods:</p>
  
  <p><strong>Method 1:</strong> Save the learnt dictionary every 100 iterations, and record the error. For 500 iterations, this gives us 5 runs of 100 iterations each. After each run, I compute the error, then use the currently learnt dictionary as an initialization for the next run. </p>
  
  <pre><code># Fit an initial dictionary, V, as a first run
  dico = MiniBatchDictionaryLearning(n_components = 100,
                                     alpha = 2,
                                     n_iter = 100,
                                     transform_algorithm='omp')
  dl = dico.fit(patches)
  V = dl.components_
  
  # Now do another 4 runs.
  # Note the warm restart parameter, dict_init = V.
  for i in range(n_runs):
      print("Run %s..." % i, end = "")
      dico = MiniBatchDictionaryLearning(n_components = 100,
                                         alpha = 2,
                                         n_iter = n_iterations,
                                         transform_algorithm='omp',
                                         dict_init = V)
      dl = dico.fit(patches)
      V = dl.components_
  
      img_r = reconstruct_image(dico, V, patches)
      err = np.sqrt(np.sum((img - img_r)**2))
      print("Err = %s" % err)
  </code></pre>
  
  <p>Problem: The error isn't decreasing, and was pretty high. The dictionary wasn't learnt very well either. </p>
  
  <p><strong>Method 2</strong>: Cut the input data X into, say, 500 batches, and do partial fitting, using the <code>partial_fit()</code> method.</p>
  
  <pre><code>batch_size = 500
  n_batches = X.shape[0] // batch_size
  print(n_batches) # 596
  
  for iternum in range(n_batches):
      batch = patches[iternum*batch_size : (iternum+1)*batch_size]
      V = dico.partial_fit(batch)
  </code></pre>
  
  <p>Problem: this seems to take about 5000 times longer. </p>
  
  <p>I'd like to know if there's a way to retrieve the error over the fitting process?</p>