title: Swapping IDs & Python Performance
body: <p>I'm hoping I can get help making my code run more efficiently. The purpose of my code is to take out the first ID (RUID) and replace it with a de-identified ID (RESPID) based on a key file of ids. The input data file is a large tab-delimited text file at about 2.5GB. The data is very wide, each row has thousands of columns. I have a function that works, but on the actual data it is incredibly slow. My first file has been running for 4 days and is only at 1.4GB. I don't know which part of my code is the most problematic, but I suspect it is where I build the row back together and write each row individually. Any advice on how to improve this would be greatly appreciated, 4 days is way too long for processing! Thank you!</p>
  
  <pre><code>def swap():
  #input files
  infile1 = open(r"Z:\ped_test.txt", 'rb')
  keyfile = open(r"Z:\ruid_respid_test.txt", 'rb')
  
  #output file
  outfile=open(r"Z:\ped_testRESPID.txt", 'wb')
  # create dictionary of RUID-RESPID 
  COLUMN = 1 #Column containing RUID
  RESPID={}
  for k in keyfile:
      kList = k.rstrip('\r\n').split('\t')
      if kList[0] not in RESPID and kList[0] != "":
          RESPID[kList[0]]=kList[1]
  #print RESPID
  print "creating RESPID-RUID xwalk dictionary is done"
  
  print "Start creating new file"
  print str(datetime.datetime.now())
  count=0
  for line in infile1:
   #if not re.match('#', line): #if there is a header     
      sline = line.split()
      #slen = len(sline)
      RUID = sline[COLUMN]
      #print RUID
      C0 = sline[0]
      #print C0
      DAT=sline[2:]
  
      for key in RESPID:
          if key==RUID:
              NewID=RESPID[key]
      row=str(C0+'\t'+NewID)
      for a in DAT:
          row=row+'\t'+a
      #print row
  outfile.write(row)
  outfile.write('\n')
  
  infile1.close()
  keyfile.close()
  outfile.close()
  
  print "All Done: RESPID replacement is complete"
  print str(datetime.datetime.now())
  </code></pre>