title: Loading large arrays from file, numpy slower than list appending. Where is the bottle neck?
body: <p>I have a very large data .dat file that I am loading into a list, or numpy array.  The file contains columns of values with the last column being a multiplier of the number of times that value should be loaded into the array. Thus, when loading it, a second sub loop needs to be ran to take that into account.  This was done to make the total file size smaller, as they were getting well over 1Gb in some cases.</p>
  
  <p>I did some testing with numpy pre allocated zeros arrays vs appending to a simply list and found the appending was faster.  Although, in numerous postings this should not be the case.  Where is the bottle neck?</p>
  
  <pre><code>    if True:
              startTime1 = time.clock()
              ## Numpy way to load up the data
              dataAry = np.zeros(TotalSamples)
              multiColAry = np.loadtxt(filename,skiprows=2,                           usecols=(columNum,lastColLoc))
              latestLoc = 0               
              for i in range(0,multiColAry.shape[0]):
                  curData = multiColAry[i][0]
                  timesBeenHere = int(multiColAry[i][2])
                  for j in range(0,timesBeenHere):
                      dataAry[latestLoc] = curData                        
                      latestLoc+=1
              endTime1 = time.clock()
              totalTime = (endTime1-startTime1) # in seconds
              totalTimeString = timeString(totalTime)
  
          if True:
              #Old string parsing directly version
              startTime1 = time.clock()
              totalAccepted = 0
              f = open(filename,'r')
              dataAry = []
              line = 'asdf'
              while line!='':
                  line = f.readline()
                  if line!='':
                      dataLineCols = line.split()
                      dataValue = float(dataLineCols[columNum])
                      timesBeenHere = float(dataLineCols[-1])
                      for j in range(0,int(timesBeenHere)):
                          totalAccepted+=1
                          dataAry.append(dataValue)
  
              f.close()
              endTime1 = time.clock()
              totalTime = (endTime1-startTime1) # in seconds
              totalTimeString = timeString(totalTime)
  </code></pre>
  
  <p>Thanks for any comments/suggestions.</p>