title: Finding document frequency using Python
body: <p>Hey everyone I know that this has been asked a couple times here already but I am having a hard time finding document frequency using python. I am trying to find TF-IDF then find the cosin scores between them and a query but am stuck at finding document frequency. This is what I have so far: </p>
  
  <pre><code>#includes
  import re
  import os
  import operator
  import glob
  import sys
  import math
  from collections import Counter
  
  #number of command line argument checker
  if len(sys.argv) != 3:
      print 'usage: ./part3_soln2.py "path to folder in quotation marks" query.txt'
      sys.exit(1)
  
  #Read in the directory to the files
      path = sys.argv[1]
  
  #Read in the query
  y = sys.argv[2]
  querystart = re.findall(r'\w+', open(y).read().lower())
  query = [Z for Z in querystart]
  Query_vec = Counter(query)
  print Query_vec
  
  #counts total number of documents in the directory
  doccounter = len(glob.glob1(path,"*.txt"))
  
  if os.path.exists(path) and os.path.isfile(y):
      word_TF = []
      word_IDF = {}
      TFvec = []
      IDFvec = []
  
      #this is my attempt at finding IDF
      for filename in glob.glob(os.path.join(path, '*.txt')):
  
          words_IDF = re.findall(r'\w+', open(filename).read().lower())
  
          doc_IDF = [A for A in words_IDF if len(A) &gt;= 3 and A.isalpha()]
  
          word_IDF = doc_IDF
  
          #psudocode!! 
          """
          for key in word_idf:
              if key in word_idf:
                  word_idf[key] =+1
              else:
                  word_idf[key] = 1
  
      print word_IDF
      """ 
  
      #goes to that directory and reads in the files there
      for filename in glob.glob(os.path.join(path, '*.txt')):
  
          words_TF = re.findall(r'\w+', open(filename).read().lower())
  
          #scans each document for words greater or equal to 3 in length
          doc_TF = [A for A in words_TF if len(A) &gt;= 3 and A.isalpha()]
  
          #this assigns values to each term this is my TF for each vector
          TFvec = Counter(doc_TF)
  
          #weighing the Tf with a log function
          for key in TFvec: 
              TFvec[key] = 1 + math.log10(TFvec[key])
  
  
      #placed here so I dont get a command line full of text  
      print TFvec 
  
  #Error checker
  else:
      print "That path does not exist"
  </code></pre>
  
  <p>I am using python 2 and so far I don't really have any idea how to count how many documents a term appears in. I can find the total number of documents but I am really stuck on finding the number of documents a term appears in. I was just going to create one large dictionary that held all of the terms from all of the documents that could be fetched later when a query needed those terms. Thank you for any help you can give me.</p>