title: How to utilize all cores with python multiprocessing
body: <p>I have been fiddling with Python's <code>multiprocessing</code> functionality for upwards of an hour now, trying to parallelize a rather complex graph traversal function using <code>multiprocessing.Process</code> and <code>multiprocessing.Manager</code>:</p>
  <pre><code>import networkx as nx
  import csv
  import time 
  from operator import itemgetter
  import os
  import multiprocessing as mp
  
  cutoff = 1
  
  exclusionlist = [&quot;cpd:C00024&quot;]
  
  DG = nx.read_gml(&quot;KeggComplete.gml&quot;, relabel=True)
  
  for exclusion in exclusionlist:
      DG.remove_node(exclusion)
  
  # checks if 'memorizedPaths exists, and if not, creates it
  fn = os.path.join(os.path.dirname(__file__),
                    'memorizedPaths' + str(cutoff+1))
  if not os.path.exists(fn):
      os.makedirs(fn)
  
  manager = mp.Manager()
  memorizedPaths = manager.dict()
  filepaths = manager.dict()
  degreelist = sorted(DG.degree_iter(),
                      key=itemgetter(1),
                      reverse=True)
  
  def _all_simple_paths_graph(item, DG, cutoff, memorizedPaths, filepaths):
      source = item[0]
      uniqueTreePaths = []
  
      if cutoff &lt; 1:
          return
  
      visited = [source]
      stack = [iter(DG[source])]
  
      while stack:
          children = stack[-1]
          child = next(children, None)
  
          if child is None:
              stack.pop()
              visited.pop()
          elif child in memorizedPaths:
              for path in memorizedPaths[child]:
                  newPath = (tuple(visited) + tuple(path))
                  if (len(newPath) &lt;= cutoff) and
                      (len(set(visited) &amp; set(path)) == 0):
                      uniqueTreePaths.append(newPath)
              continue
          elif len(visited) &lt; cutoff:
              if child not in visited:
                  visited.append(child)
                  stack.append(iter(DG[child]))
  
                  if visited not in uniqueTreePaths:
                      uniqueTreePaths.append(tuple(visited))
          else: # len(visited) == cutoff:
              if (visited not in uniqueTreePaths) and
                  (child not in visited):
                  uniqueTreePaths.append(tuple(visited + [child]))
              stack.pop()
              visited.pop()
      # writes the absolute path of the node path file into the hash table
      filepaths[source] = str(fn) + &quot;/&quot; + str(source) + &quot;path.txt&quot;
      with open (filepaths[source], &quot;wb&quot;) as csvfile2:
          writer = csv.writer(csvfile2, delimiter=&quot; &quot;, quotechar=&quot;|&quot;)
          for path in uniqueTreePaths:
              writer.writerow(path)
  
      memorizedPaths[source] = uniqueTreePaths
  
  ############################################################################
  
  if __name__ == '__main__':
      start = time.clock()
  
      for item in degreelist:
          test = mp.Process(target=_all_simple_paths_graph,
                            args=(DG, cutoff, item, memorizedPaths, filepaths))
          test.start()
          test.join()
  
  end = time.clock()
  print (end-start)
  </code></pre>
  <p>Currently - though luck and magic - it works (sort of). My problem is I'm only using 12 of my 24 cores.</p>
  <p>Can someone explain why this might be the case? Perhaps my code isn't the best multiprocessing solution, or is it a feature of my architecture <em>Intel Xeon CPU E5-2640 @ 2.50GHz x18 running on Ubuntu 13.04 x64</em>?</p>
  <p><strong>EDIT:</strong></p>
  <p>I managed to get:</p>
  <pre><code>p = mp.Pool()
  for item in degreelist:
      p.apply_async(_all_simple_paths_graph,
                    args=(DG, cutoff, item, memorizedPaths, filepaths))
  p.close()
  p.join()
  </code></pre>
  <p>Working, however, it's <strong>VERY SLOW</strong>! So I assume I'm using the wrong function for the job. hopefully it helps clarify exactly what I'm trying to accomplish!</p>
  <p>EDIT2: <code>.map</code> attempt:</p>
  <pre><code>partialfunc = partial(_all_simple_paths_graph,
                        DG=DG,
                        cutoff=cutoff,
                        memorizedPaths=memorizedPaths,
                        filepaths=filepaths)
  p = mp.Pool()
  for item in processList:
      processVar = p.map(partialfunc, xrange(len(processList)))   
  p.close()
  p.join()
  </code></pre>
  <p>Works, is slower than singlecore. Time to optimize!</p>