title: Python multiprocessing - Debugging OSError: [Errno 12] Cannot allocate memory
body: <p>I'm facing the following issue. I'm trying to parallelize a function that updates a file, but I cannot start the <code>Pool()</code> because of an <code>OSError: [Errno 12] Cannot allocate memory</code>. I've started looking around on the server, and it's not like I'm using an old, weak one/out of actual memory.
  See <code>htop</code>:
  <a href="https://i.stack.imgur.com/UEWjc.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UEWjc.png" alt="enter image description here"></a>
  Also, <code>free -m</code> shows I have plenty of RAM available in addition to the ~7GB of swap memory:
  <a href="https://i.stack.imgur.com/Nuxjx.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Nuxjx.png" alt="enter image description here"></a>
  And the files I'm trying to work with aren't that big either. I'll paste my code (and the stack trace) below, there, the sizes are as follows:</p>
  
  <p>The <code>predictionmatrix</code> dataframe used takes up ca. 80MB according to <code>pandasdataframe.memory_usage()</code>
  The file <code>geo.geojson</code> is 2MB</p>
  
  <p>How do I go about debugging this? What can I check and how? Thank you for any tips/tricks!</p>
  
  <p>Code:</p>
  
  <pre><code>def parallelUpdateJSON(paramMatch, predictionmatrix, data):
      for feature in data['features']: 
          currentfeature = predictionmatrix[(predictionmatrix['SId']==feature['properties']['cellId']) &amp; paramMatch]
          if (len(currentfeature) &gt; 0):
              feature['properties'].update({"style": {"opacity": currentfeature.AllActivity.item()}})
          else:
              feature['properties'].update({"style": {"opacity": 0}})
  
  def writeGeoJSON(weekdaytopredict, hourtopredict, predictionmatrix):
      with open('geo.geojson') as f:
          data = json.load(f)
      paramMatch = (predictionmatrix['Hour']==hourtopredict) &amp; (predictionmatrix['Weekday']==weekdaytopredict)
      pool = Pool()
      func = partial(parallelUpdateJSON, paramMatch, predictionmatrix)
      pool.map(func, data)
      pool.close()
      pool.join()
  
      with open('output.geojson', 'w') as outfile:
          json.dump(data, outfile)
  </code></pre>
  
  <p>Stack Trace:</p>
  
  <pre><code>---------------------------------------------------------------------------
  OSError                                   Traceback (most recent call last)
  &lt;ipython-input-428-d6121ed2750b&gt; in &lt;module&gt;()
  ----&gt; 1 writeGeoJSON(6, 15, baseline)
  
  &lt;ipython-input-427-973b7a5a8acc&gt; in writeGeoJSON(weekdaytopredict, hourtopredict, predictionmatrix)
       14     print("Start loop")
       15     paramMatch = (predictionmatrix['Hour']==hourtopredict) &amp; (predictionmatrix['Weekday']==weekdaytopredict)
  ---&gt; 16     pool = Pool(2)
       17     func = partial(parallelUpdateJSON, paramMatch, predictionmatrix)
       18     print(predictionmatrix.memory_usage())
  
  /usr/lib/python3.5/multiprocessing/context.py in Pool(self, processes, initializer, initargs, maxtasksperchild)
      116         from .pool import Pool
      117         return Pool(processes, initializer, initargs, maxtasksperchild,
  --&gt; 118                     context=self.get_context())
      119 
      120     def RawValue(self, typecode_or_type, *args):
  
  /usr/lib/python3.5/multiprocessing/pool.py in __init__(self, processes, initializer, initargs, maxtasksperchild, context)
      166         self._processes = processes
      167         self._pool = []
  --&gt; 168         self._repopulate_pool()
      169 
      170         self._worker_handler = threading.Thread(
  
  /usr/lib/python3.5/multiprocessing/pool.py in _repopulate_pool(self)
      231             w.name = w.name.replace('Process', 'PoolWorker')
      232             w.daemon = True
  --&gt; 233             w.start()
      234             util.debug('added worker')
      235 
  
  /usr/lib/python3.5/multiprocessing/process.py in start(self)
      103                'daemonic processes are not allowed to have children'
      104         _cleanup()
  --&gt; 105         self._popen = self._Popen(self)
      106         self._sentinel = self._popen.sentinel
      107         _children.add(self)
  
  /usr/lib/python3.5/multiprocessing/context.py in _Popen(process_obj)
      265         def _Popen(process_obj):
      266             from .popen_fork import Popen
  --&gt; 267             return Popen(process_obj)
      268 
      269     class SpawnProcess(process.BaseProcess):
  
  /usr/lib/python3.5/multiprocessing/popen_fork.py in __init__(self, process_obj)
       18         sys.stderr.flush()
       19         self.returncode = None
  ---&gt; 20         self._launch(process_obj)
       21 
       22     def duplicate_for_child(self, fd):
  
  /usr/lib/python3.5/multiprocessing/popen_fork.py in _launch(self, process_obj)
       65         code = 1
       66         parent_r, child_w = os.pipe()
  ---&gt; 67         self.pid = os.fork()
       68         if self.pid == 0:
       69             try:
  
  OSError: [Errno 12] Cannot allocate memory
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>According to @robyschek's solution, I've updated my code to:</p>
  
  <pre><code>global g_predictionmatrix 
  
  def worker_init(predictionmatrix):
      global g_predictionmatrix
      g_predictionmatrix = predictionmatrix    
  
  def parallelUpdateJSON(paramMatch, data_item):
      for feature in data_item['features']: 
          currentfeature = predictionmatrix[(predictionmatrix['SId']==feature['properties']['cellId']) &amp; paramMatch]
          if (len(currentfeature) &gt; 0):
              feature['properties'].update({"style": {"opacity": currentfeature.AllActivity.item()}})
          else:
              feature['properties'].update({"style": {"opacity": 0}})
  
  def use_the_pool(data, paramMatch, predictionmatrix):
      pool = Pool(initializer=worker_init, initargs=(predictionmatrix,))
      func = partial(parallelUpdateJSON, paramMatch)
      pool.map(func, data)
      pool.close()
      pool.join()
  
  
  def writeGeoJSON(weekdaytopredict, hourtopredict, predictionmatrix):
      with open('geo.geojson') as f:
          data = json.load(f)
      paramMatch = (predictionmatrix['Hour']==hourtopredict) &amp; (predictionmatrix['Weekday']==weekdaytopredict)
      use_the_pool(data, paramMatch, predictionmatrix)     
      with open('trentino-grid.geojson', 'w') as outfile:
          json.dump(data, outfile)
  </code></pre>
  
  <p>And I still get the same error. Also, according to the <a href="https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map" rel="noreferrer">documentation</a>, <code>map()</code> should divide my <code>data</code> into chunks, so I don't think it should replicate my 80MBs rownum times. I may be wrong though... :)
  Plus I've noticed that if I use smaller input (~11MB instead of 80MB) I don't get the error. So I guess I'm trying to use too much memory, but I can't imagine how it goes from 80MB to something 16GBs of RAM can't handle.</p>