title: Read and write large file with python
body: <p>I have a large <code>.txt</code> file with more than 24,000,000 lines. Now I'd like to do a word count, that is, count each word and its corresponding occurrence and record them to a new file. Here is the code I tried to run:</p>
  
  <pre><code>import gensim
  class Corpus(gensim.corpora.TextCorpus): 
      def count_tokens(self):
          word_count = 0
          for text in self.get_texts():
              word_count += len(text)
          return word_count
      def get_texts(self): 
          for filename in self.input: 
              yield open(filename).read().split()
  
  def main():
      corpus = Corpus(['somefile.txt'])
      word_count = corpus.count_tokens()
      text = open('somefile.txt').read().split()
      with open('path_to_output', 'w') as f:
          for word, _ in corpus.dictionary.token2id.items():
              num_occur = text.count(word)
              f.write('%s %d\n' % (word, num_occur))
  
  if __name__  == '__main__':
      main()
  </code></pre>
  
  <p>And the server hang... I wonder if there is other sufficient way to do so or any improvement I can make? How do you read and write really large file with python?</p>