title: Optimizing find and replace over large files in Python
body: <p>I am a <strong>complete beginner</strong> to Python or any serious programming language for that matter. I finally got a prototype code to work but I think it will be too slow.</p>
  
  <p>My goal is to find and replace some Chinese characters across all files (they are csv) in a directory with integers as per a csv file I have. The files are nicely numbered by year-month, for example 2000-01.csv, and will be the only files in that directory.</p>
  
  <p>I will be looping across about 25 files that are in the neighborhood of 500mb each (and about a million lines). The dictionary I will be using will have about 300 elements and I will be changing unicode (Chinese character) to integers. I tried with a test run and, assuming everything scales up linearly (?), it looks like it would take about a week for this to run.</p>
  
  <p>Thanks in advance. Here is my code (don't laugh!):</p>
  
  <pre><code># -*- coding: utf-8 -*-
  
  import os, codecs
  
  dir = "C:/Users/Roy/Desktop/test/"
  
  Dict = {'hello' : 'good', 'world' : 'bad'}
  
  for dirs, subdirs, files in os.walk(dir):
      for file in files:
          inFile = codecs.open(dir + file, "r", "utf-8")
          inFileStr = inFile.read()
          inFile.close()
          inFile = codecs.open(dir + file, "w", "utf-8")
          for key in Dict:
              inFileStr = inFileStr.replace(key, Dict[key])
          inFile.write(inFileStr)
          inFile.close()
  </code></pre>