title: multiprocessing global variable memory copying
body: <p>I am running a program which loads 20 GB data to the memory at first. Then I will do N (> 1000) independent tasks where each of them may use (read only) part of the 20 GB data. I am now trying to do those tasks via multiprocessing. However, as <a href="https://stackoverflow.com/questions/11055303/python-multiprocessing-global-variable-updates-not-returned-to-parent">this answer</a> says, the entire global variables are copied for each process. In my case, I do not have enough memory to perform more than 4 tasks as my memory is only 96 GB. I wonder if there is any solution to this kind of problem so that I can fully use all my cores without consuming too much memory.  </p>