title: Appending frequencies from a corpus to each token in a tweet
body: <p>I am working with tweet data which is POS-tagged with the NLTK POS-tagger. 
  My tokens look like: </p>
  
  <pre><code>[['wasabi', 'NN'], 
  ['juice', 'NN']]
  </code></pre>
  
  <p>I also have the American National Corpus Frequencies, a list of words, POS-tags and their frequencies. I would like to look up the word and pos-tag from the tokens, and if it's found, append the frequency from the ANC to the token. </p>
  
  <p>Excellent suggestions from SO have helped, but I found that several of the tokens were not getting frequencies appended (probably because the NLTK tagger is pretty inaccurate, e.g. calling 'silent' a noun, instead of an adjective) and when I tried to append just the frequency, I kept getting a key error, because NLTK had tagged 'jill' as NN, not NNP. </p>
  
  <p>Finally, I decided to take the first frequency if the word is found. The issue now is that I'm getting all the frequencies for occurrences of the word. I just want the first one, so the output would be: </p>
  
  <pre><code>[['wasabi', 'NN', '5'], 
  ['juice', 'NN', '369']]
  </code></pre>
  
  <p>Code,</p>
  
  <pre class="lang-py prettyprint-override"><code>with open('ANC-all-count.txt', 'r', errors='ignore') as f:
      freqs = csv.reader(f, delimiter='\t')
  
      freqs = {}
      for word, pos, f in freq_list:
          if word not in freqs: freqs[word] = {}
          freqs[word][pos] = f
  
          for i, (word, pos) in enumerate(tokens):
              if word not in freqs: 
                  tokens[i].append(0)
                  continue
              if pos not in freqs[word]:
                  tokens[i] = [tokens[i][0:2]]
                  single_token = tokens[i][0]
                  if single_token[0] in freqs:
                      tokens[i].append(freqs[word].values())
                  continue
              tokens[i].append(freqs[word][pos])
  </code></pre>