title: text content analyzer in python
body: <p>I  created a text content analyzer in python that analyzes input from a file and output</p>
  
  <ol>
  <li>The total word count</li>
  <li>The count of unique words</li>
  <li>The number of sentences</li>
  </ol>
  
  <p>Here is the code :</p>
  
  <pre><code>import re
  import string
  import os
  import sys
  
  def function(s):
      return re.sub("[%s]" % re.escape(string.punctuation), '', s.lower())
  
  def main():
      words_list = []
  
      with open(sys.argv[1], "r") as f:
          for line in f:
              words_list.extend(line.split())
  
      print "Total word count:", len(words_list)
  
      new_words = map(function, words_list)
  
      print "Unique words:", len(set(new_words))
  
      nb_sentence = 0
      for word in words_list:
          if re.search(r'[.!?][' "'" '"]*', word):
              nb_sentence += 1
  
      print "Sentences:", nb_sentence
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>I am now trying to calculate the average sentence length in words, find often used phrases (a phrase of 3 or more words used over 3 times), and make a list of words used, in order of descending frequency. Could anyone help?</p>