title: NLTK package to estimate the (unigram) perplexity
body: <p>I am trying to calculate the perplexity for the data I have. The code I am using is: </p>
  
  <pre><code> import sys
   sys.path.append("/usr/local/anaconda/lib/python2.7/site-packages/nltk")
  
  from nltk.corpus import brown
  from nltk.model import NgramModel
  from nltk.probability import LidstoneProbDist, WittenBellProbDist
  estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2)
  lm = NgramModel(3, brown.words(categories='news'), True, False, estimator)
  print lm
  </code></pre>
  
  <p>But I am receiving the error, </p>
  
  <pre><code>File "/usr/local/anaconda/lib/python2.7/site-packages/nltk/model/ngram.py", line 107, in __init__
  cfd[context][token] += 1
  TypeError: 'int' object has no attribute '__getitem__'
  </code></pre>
  
  <p>I have already performed Latent Dirichlet Allocation for the data I have and I have generated the unigrams and their respective probabilities (they are normalized as the sum of total probabilities of the data is 1).</p>
  
  <p>My unigrams and their probability looks like:</p>
  
  <pre><code>Negroponte 1.22948976891e-05
  Andreas 7.11290670484e-07
  Rheinberg 7.08255885794e-07
  Joji 4.48481435106e-07
  Helguson 1.89936727391e-07
  CAPTION_spot 2.37395965468e-06
  Mortimer 1.48540253778e-07
  yellow 1.26582575863e-05
  Sugar 1.49563800878e-06
  four 0.000207196011781
  </code></pre>
  
  <p>This is just a fragment of the unigrams file I have. The same format is followed for about 1000s of lines. The total probabilities (second column) summed gives 1. </p>
  
  <p>I am a budding programmer. This ngram.py belongs to the nltk package and I am confused as to how to rectify this. The sample code I have here is from the nltk documentation and I don't know what to do now. Please help on what I can do. Thanks in advance!</p>