title: NLTK language modeling confusion
body: <p>I want to train a language model using NLTK in python but I got into several problems.
  first of all, I don't know why my words turn into just characters as I write something like this :</p>
  
  <pre><code>s = "Natural-language processing (NLP) is an area of computer science " \
  "and artificial intelligence concerned with the interactions " \
  "between computers and human (natural) languages."
  s = s.lower();
  
  
  paddedLine = pad_both_ends(word_tokenize(s),n=2);
  
  train, vocab = padded_everygram_pipeline(2, paddedLine)
  print(list(vocab))
  lm = MLE(2);
  lm.fit(train,vocab)
  </code></pre>
  
  <p>and the printed vocab is something like this that is clearly not correct(i don't want to work with characters!),this is part of output.:</p>
  
  <pre><code>&lt;s&gt;', '&lt;', 's', '&gt;', '&lt;/s&gt;', '&lt;s&gt;', 'n', 'a', 't', 'u', 'r', 'a', 'l', '-', 'l', 'a', 'n', 'g', 'u', 'a', 'g', 'e', '&lt;/s&gt;', '&lt;s&gt;', 'p', 'r', 'o', 'c', 'e', 's', 's', 'i', 'n', 'g', '&lt;/s&gt;', '&lt;s&gt;', '(', '&lt;/s&gt;', '&lt;s&gt;', 'n', 'l', 'p', '&lt;/s&gt;', '&lt;s&gt;', ')', '&lt;/s&gt;'
  </code></pre>
  
  <p>why my input turns into characters?
  i did this work in another way but with no luck :</p>
  
  <pre><code>paddedLine = pad_both_ends(word_tokenize(s),n=2);
  #train, vocab = padded_everygram_pipeline(2, tokens)
  #train = everygrams(paddedLine,max_len = 2);
  
  train = ngrams(paddedLine,2);
  vocab = Vocabulary(paddedLine,unk_cutoff = 1);
  print(list(train))
  
  lm = MLE(2);
  lm.fit(train,vocab)
  </code></pre>
  
  <p>when i run this code my train is absolute nothing,empty! it shows me "[]" !!
  wired thing is when i comment at this line from above code:</p>
  
  <pre><code>vocab = Vocabulary(paddedLine,unk_cutoff = 1);
  </code></pre>
  
  <p>now my train data is ok and something like this that is correct :</p>
  
  <pre><code>[('&lt;s&gt;', 'natural-language'), ('natural-language', 'processing'), ('processing', '('), ('(', 'nlp'), ('nlp', ')'), (')', 'is'), ('is', 'an'), ('an', 'area'), ('area', 'of'), ('of', 'computer'), ('computer', 'science'), ('science', 'and'), ('and', 'artificial'), ('artificial', 'intelligence'), ('intelligence', 'concerned'), ('concerned', 'with'), ('with', 'the'), ('the', 'interactions'), ('interactions', 'between'), ('between', 'computers'), ('computers', 'and'), ('and', 'human'), ('human', '('), ('(', 'natural'), ('natural', ')'), (')', 'languages'), ('languages', '.'), ('.', '&lt;/s&gt;')]
  </code></pre>
  
  <p>whats wrong with it? 
  by the way, I have to say that I'm not an expert in python or NLTK and it's my first experience.
  The next question is how can I use kneser-ney smoothing or add-one smoothing on the training language model? 
  and am I doing language model training the right way?
  my training data is simple :</p>
  
  <pre><code>"Natural-language processing (NLP) is an area of computer science " \
      "and artificial intelligence concerned with the interactions " \
      "between computers and human (natural) languages."
  </code></pre>
  
  <p>thanks.</p>