title: Processes not parallel
body: <p>I'm working on a classification tool using dbpedia dumps and I ran into a performance problem (multiple days of processing). I want to classify each wikipedia article -- in short I'm merging an article with similar articles and combine outputs from multiple classifiers. I wanted to leverage my pc (i7-6700K) and run the program on multiple cores/proceses but I cant seem to get it working. I ended up with multiple processes but only one is running at a time.</p>
  
  <p>I'm running ubuntu as and windows subsystem.</p>
  
  <p><a href="https://i.stack.imgur.com/xr6Gj.png" rel="nofollow noreferrer">task manager</a></p>
  
  <p>The input file looks like this:</p>
  
  <pre class="lang-none prettyprint-override"><code># started 2016-06-16T01:23:53Z
  &lt;http://dbpedia.org/resource/Achilles&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://www.w3.org/2002/07/owl#Thing&gt; .
  &lt;http://dbpedia.org/resource/An_American_in_Paris&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://www.w3.org/2002/07/owl#Thing&gt; .
  &lt;http://dbpedia.org/resource/Actrius&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://dbpedia.org/ontology/Film&gt; .
  &lt;http://dbpedia.org/resource/Animalia_(book)&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://dbpedia.org/ontology/Book&gt; .
  &lt;http://dbpedia.org/resource/Agricultural_science&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://www.w3.org/2002/07/owl#Thing&gt; .
  &lt;http://dbpedia.org/resource/Alain_Connes&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://dbpedia.org/ontology/Scientist&gt; .
  &lt;http://dbpedia.org/resource/Allan_Dwan&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;http://dbpedia.org/ontology/Person&gt; .
  </code></pre>
  
  <p>Here is my code:</p>
  
  <pre><code>def processWrapper(self, resourceInput, chunkStart, chunkSize):
      results = []
      with open(resourceInput, 'rb') as f:
          f.seek(chunkStart)
          lines = f.read(chunkSize).splitlines()
          logger.info(lines)
          for line in lines:
              if line.strip().startswith(b"#"):
                  continue
  
              x = line.strip().split(b" ")
              logger.info(line.strip())
              resource = x[0][1:-1].decode("utf-8")
              if regSearch(r"__\d+", resource):
                  continue
              logger.info(resource)
              result = self.resultTresholding(self.aggregateClassfierResults(resource), self.tresholdStrategy, self.threshold)
              finalType = self.finalTypeSelection(self.finalTypeSelectionStrategy, self.tresholdStrategy, result)
              results.append(tuple(result, finalType))
          return results
  
  def chunkify(self, fname, size=1024*1024):
      fileEnd = os.path.getsize(fname)
      print(fileEnd)
      with open(fname, 'rb') as f:
          chunkEnd = f.tell()
          while True:
              chunkStart = chunkEnd
              f.seek(size, 1)
              f.readline()
              chunkEnd = f.tell()
              yield chunkStart, chunkEnd - chunkStart
              if chunkEnd &gt; fileEnd:
                  break
  
  def processResources(self, resourceInput):
      '''
  
      '''
      pool = mp.Pool(4)
      # graph = rdflib.Graph()
  
      jobs = []
  
      # create jobs
      for chunkStart, chunkSize in self.chunkify(str(resourceInput), 100):
          logger.info(f"chunkStart - {chunkStart}")
          logger.info(f"chunkSize - {chunkSize}")
          jobs.append(pool.apply_async(self.processWrapper, (resourceInput, chunkStart, chunkSize)))
  
      for job in jobs:
          for result in job.get():
              with open(s.resultFile, 'a') as g:
                  g.write(f"&lt;{result[0]}&gt; &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#type&gt; &lt;{result[1]}&gt;  .\n")
      pool.close()
  </code></pre>
  
  <p>What am I missing? This is my first time using <code>multiprocessing</code>...</p>