title: Python input/output optimisation
body: <p>I think this code takes too long to execute, so maybe there are better ways to do this. I'm not looking for an answer related to parallelising the for loops, or using more than one processor.</p>
  
  <p>What I'm trying to do is to read values from "file" using "np.genfromtxt(file)". I have 209*500*16 of these files. I want to extract the minimum value of the highest 1000 values of the 209 loop, and putting these 500 values in 16 different files. If the files are missing or the data hasn't the adequate size, the info is written to the "missing_all" file.</p>
  
  <p>The questions are:</p>
  
  <ol>
  <li><p>Is this the best method to open a file?</p></li>
  <li><p>Is this the best method to write to files?</p></li>
  <li><p>How can I make this code faster?</p></li>
  </ol>
  
  <p>Code:</p>
  
  <pre><code>import numpy as np
  import os.path
  
  output_filename2 = '/home/missing_all.txt' 
  target2          = open(output_filename2, 'w')
  
  for w in range(16):
      group           = 1200 + 50*w
      output_filename = '/home/veto_%s.txt' %(group)
      target          = open(output_filename, 'w')
      for z in range(1,501):
          sig_b = np.zeros((209*300))
          y     = 0
          for index in range(1,210):
              file                 = '/home/BandNo_%s_%s/%s_209.dat' %(group,z,index)
              if not os.path.isfile(file):
                  sig_b[y:y+300]   = 0
                  y                = y + 300
                  target2.write('%s %s %s\n' % (group,z,index))
                  continue
              data                 = np.genfromtxt(file)
              if (data.shape[0] &lt; 300):
                  sig_b[y:y+300]   = 0
                  y                = y + 300
                  target2.write('%s %s %s\n' % (group,z,index))
                  continue
              sig_b[y:y+300]       = np.sort(data[:,4])[::-1][0:300]
              y                    = y + 300  
          sig_b          = np.sort(sig_b[:])[::-1][0:1000]   
          target.write('%s\n' % (sig_b[-1]))
  </code></pre>