title: Implementation of Naive Bayes - accuracy issues
body: <p><strong>EDIT: The correct version of the code which works can be found at :</strong> 
  <a href="https://github.com/a7x/NaiveBayes-Classifier" rel="nofollow">https://github.com/a7x/NaiveBayes-Classifier</a></p>
  
  <p>I used data from <a href="http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;doc=exercises/ex6/ex6.html" rel="nofollow">openClassroom</a> and started working on a small version of Naive Bayes in Python. Steps were the usual training and then prediction . I have a few questions and want to know why the accuracy is quite bad.</p>
  
  <ol>
  <li><p>For training, I calculated the log likelihood by the formula  : </p>
  
  <blockquote>
    <p>log( P ( word  | spam ) +1 ) /( spamSize + vocabSize .)</p>
  </blockquote>
  
  <p>My question is: why did we add the <code>vocabSize</code> in this case :( and is this the correct way of going about it? Code used is below: </p>
  
  <pre><code>#This is for training.     Calculate all probabilities and store them in a vector. Better to store it in a file  for easier access 
  from __future__ import division
  import sys,os
  ''' 
  1. The spam and non-spam is already 50%  . So they by default are 0.5
  2. Now we need to calculate probability of each word    , in spam and non-spam separately
    2.1  we can make two dictionaries, defaultdicts basically,  for spam and non-spam 
    2.2 When time comes to calculate probabilities, we just need to substitute values
  '''
  from collections import *
  from math import *
  
  spamDict = defaultdict(int)
  nonspamDict = defaultdict(int)
  spamFolders = ["spam-train"]
  nonspamFolders = ["nonspam-train"]
  path = sys.argv[1] #Base path
  spamVector = open(sys.argv[2],'w') #WRite all spam values into this 
  nonspamVector = open(sys.argv[3],'w') #Non-spam values
  
  #Go through all files in spam and  iteratively add values
  spamSize = 0
  nonspamSize = 0
  vocabSize = 264821
  for f in os.listdir(os.path.join(path,spamFolders[0])):
      data = open(os.path.join(path,spamFolders[0],f),'r')
  
      for line in data:
          words = line.split(" ")
          spamSize = spamSize + len(words)
          for w in words:
              spamDict[w]+=1
  
  for f in os.listdir(os.path.join(path,nonspamFolders[0])):
      data = open(os.path.join(path,nonspamFolders[0],f),'r')
      for line in data:
          words = line.split(" ")
          nonspamSize = nonspamSize + len(words)
          for w in words:
  
              nonspamDict[w]+=1
  logProbspam = {}
  logProbnonSpam = {} #This is to store the log probabilities
  for k in spamDict.keys():
      #Need to calculate P(x | y = 1)
  
      numerator =  spamDict[k] + 1  # Frequency
      print 'Word',k,' frequency',spamDict[k]
      denominator = spamSize + vocabSize
      p = log(numerator/denominator)
      logProbspam[k] = p
  for k in nonspamDict.keys():
      numerator = nonspamDict[k] + 1 #frequency
      denominator = nonspamSize + vocabSize
      p = log(numerator/denominator)
      logProbnonSpam[k] = p
  
  for k in logProbnonSpam.keys():
      nonspamVector.write(k+" "+str(logProbnonSpam[k])+"\n")
  for k in logProbspam.keys():
      spamVector.write(k+" "+str(logProbspam[k])+"\n")
  </code></pre></li>
  <li><p>For prediction, I just took a mail , split it into words, added all the probabilities, separately for 
  spam/non-spam, and multiplied them by 0.5. Whichever was higher was the class label. Code is below: </p>
  
  <p><a href="http://pastebin.com/8Y6Gm2my" rel="nofollow">http://pastebin.com/8Y6Gm2my</a> ( Stackoverflow was again playing games for some reason :-/) </p></li>
  </ol>
  
  <p><strong>EDIT : I have removed the spam = spam + 1 thing. Instead, I just ignore those words</strong> </p>
  
  <p>Problem : My accuracy is quite bad .  As noted below.</p>
  
  <pre><code>    No of files in spam is 130
      No. of spam in  ../NaiveBayes/spam-test  is  53  no. of non-spam 77
      No of files in non-spam is 130
      No. of spam in  ../NaiveBayes/nonspam-test/  is  6  no. of non-spam 124
  </code></pre>
  
  <p>Please tell me where all I am going wrong. I think an accuracy of less than 50% means there must be some glaring error(s) in the implementation.</p>