title: Python multiprocessing memory usage
body: <p>I have writen a program that can be summarized as follows:</p>
  <pre><code>def loadHugeData():
      #load it
      return data
  
  def processHugeData(data, res_queue):
      for item in data:
          #process it
          res_queue.put(result)
      res_queue.put(&quot;END&quot;)
  
  def writeOutput(outFile, res_queue):
      with open(outFile, 'w') as f
          res=res_queue.get()
          while res!='END':
              f.write(res)
              res=res_queue.get()
  
  res_queue = multiprocessing.Queue()
  
  if __name__ == '__main__':
      data=loadHugeData()
      p = multiprocessing.Process(target=writeOutput, args=(outFile, res_queue))
      p.start()
      processHugeData(data, res_queue)
      p.join()
  </code></pre>
  <p>The real code (especially <code>writeOutput()</code>) is a lot more complicated. <code>writeOutput()</code> only uses these values that it takes as its arguments (meaning it does not reference <code>data</code>)</p>
  <p>Basically it loads a huge dataset into memory and processes it. Writing of the output is delegated to a sub-process (it writes into multiple files actually and this takes a lot of time).
  So each time one data item gets processed it is sent to the sub-process trough res_queue which in turn writes the result into files as needed.</p>
  <p>The sub-process does not need to access, read or modify the data loaded by <code>loadHugeData()</code> in any way. The sub-process only needs to use what the main process sends it trough <code>res_queue</code>. And this leads me to my problem and question.</p>
  <p>It seems to me that the sub-process gets its own copy of the huge dataset (when checking memory usage with <code>top</code>). Is this true? And if so then how can i avoid id (using double memory essentially)?</p>
  <p>I am using Python 2.6 and program is running on linux.</p>