title: Python, process a large text file in parallel
body: <p>Samples records in the data file (SAM file):</p>
  
  <pre><code>M01383  0  chr4  66439384  255  31M  *  0  0  AAGAGGA GFAFHGD  MD:Z:31 NM:i:0
  M01382  0  chr1  241995435  255 31M  *  0  0  ATCCAAG AFHTTAG  MD:Z:31 NM:i:0
  ......
  </code></pre>
  
  <ul>
  <li>The data files are line-by-line based</li>
  <li>The size of the data files are varies from 1G - 5G.</li>
  </ul>
  
  <p>I need to go through the record in the data file line by line, get a particular value (e.g. 4th value, 66439384) from each line, and pass this value to another function for processing. Then some results counter will be updated.</p>
  
  <p>the basic workflow is like this:</p>
  
  <pre><code># global variable, counters will be updated in search function according to the value passed. 
  counter_a = 0    
  counter_b = 0
  counter_c = 0
  
  open textfile:
      for line in textfile:
          value = line.split()[3]
          search_function(value)    # this function takes abit long time to process
  
  def search_function (value):
      some conditions checking:
          update the counter_a or counter_b or counter_c
  </code></pre>
  
  <p>With single process code and about 1.5G data file, it took about 20 hours to run through all the records in one data file. I need much faster code because there are more than 30 of this kind data file. </p>
  
  <p>I was thinking to process the data file in N chunks in parallel, and each chunk will perform above workflow and update the global variable (counter_a, counter_b, counter_c) simultaneously. But I don't know how to achieve this in code, or wether this will work. </p>
  
  <p>I have access to a server machine with:  24 processors and around 40G RAM. </p>
  
  <p>Anyone could help with this? Thanks very much. </p>