title: Counting bigrams real fast (with or without multiprocessing) - python
body: <p>Given the <code>big.txt</code> from <a href="http://norvig.com/big.txt" rel="nofollow noreferrer">norvig.com/big.txt</a>, the goal is to count the bigrams really fast (Imagine that I have to repeat this counting 100,000 times).</p>
  
  <p>According to <a href="https://stackoverflow.com/questions/21883108/fast-optimize-n-gram-implementations-in-python">Fast/Optimize N-gram implementations in python</a>, extracting bigrams like this would be the most optimal:</p>
  
  <pre><code>_bigrams = zip(*[text[i:] for i in range(2)])
  </code></pre>
  
  <p>And if I'm using <code>Python3</code>, the generator won't be evaluated until i materialize it with <code>list(_bigrams)</code> or some other functions that will do the same.</p>
  
  <pre><code>import io
  from collections import Counter
  
  import time
  with io.open('big.txt', 'r', encoding='utf8') as fin:
       text = fin.read().lower().replace(u' ', u"\uE000")
  
  while True: 
      _bigrams = zip(*[text[i:] for i in range(2)])
      start = time.time()
      top100 = Counter(_bigrams).most_common(100)
      # Do some manipulation to text and repeat the counting.
      text = manipulate(text, top100)      
  </code></pre>
  
  <p>But that takes around 1+ seconds per iteration, and 100,000 iterations would be too long. </p>
  
  <p>I've also tried <code>sklearn</code> CountVectorizer but the time to extract, count and get the top100 bigrams are comparable to the native python. </p>
  
  <p>Then I've experimented with some <code>multiprocessing</code>, using slight modification from <a href="https://stackoverflow.com/questions/2080660/python-multiprocessing-and-a-shared-counter">Python multiprocessing and a shared counter</a> and <a href="http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing" rel="nofollow noreferrer">http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing</a>:</p>
  
  <pre><code>from multiprocessing import Process, Manager, Lock
  
  import time
  
  class MultiProcCounter(object):
      def __init__(self):
          self.dictionary = Manager().dict()
          self.lock = Lock()
  
      def increment(self, item):
          with self.lock:
              self.dictionary[item] = self.dictionary.get(item, 0) + 1
  
  def func(counter, item):
      counter.increment(item)
  
  def multiproc_count(inputs):
      counter = MultiProcCounter()
      procs = [Process(target=func, args=(counter,_in)) for _in in inputs]
      for p in procs: p.start()
      for p in procs: p.join()
      return counter.dictionary
  
  inputs = [1,1,1,1,2,2,3,4,4,5,2,2,3,1,2]
  
  print (multiproc_count(inputs))
  </code></pre>
  
  <p>But using the <code>MultiProcCounter</code> in the bigram counting takes even longer than 1+ seconds per iteration. I've no idea why is that the case, using the dummy list of <code>int</code> example, the <code>multiproc_count</code> works perfectly.</p>
  
  <p>I've tried:</p>
  
  <pre><code>import io
  from collections import Counter
  
  import time
  with io.open('big.txt', 'r', encoding='utf8') as fin:
       text = fin.read().lower().replace(u' ', u"\uE000")
  
  while True:
      _bigrams = zip(*[text[i:] for i in range(2)])
      start = time.time()
      top100 = Counter(multiproc_count(_bigrams)).most_common(100)
  </code></pre>
  
  <p><strong>Is there any way to count the bigrams really fast in Python?</strong></p>