title: Python / Pandas / spacy - iterate over a DataFrame and count the number of pos_ tags
body: <p>i have a Pandas Dataframe with some Texts from an Author and want to do some statistical stuff with the sum of the different word types.</p>
  
  <p><strong>Dataframe - my data :</strong></p>
  
  <pre><code>&gt;&gt;&gt; data
               name                   style                                              text     year       year_dt
  number  
  0001    Demetrius                       D   Demetrius an der russischen Grenze Er ist vo...     1805    1805-01-01
  0002    Der versöhnte Menschenfeind     D   Der versöhnte Menschenfeind -Fragment Gegend...     1790    1790-01-01
  0003    Die Braut von Messina           D   Die Braut von Messina oder die feindlichen B...     1803    1803-01-01
  </code></pre>
  
  <p>Some months ago i wrote a function that iterates from line to line of the df, take the name and the content of "the book", made a pos tagging from spacy, and count the number of nouns, adjectives and verbs for the beginning. after that, the number is stored in a new column.</p>
  
  <p><strong>my function:</strong></p>
  
  <pre><code>import spacy
  from spacy.lang.de import German
  from collections import defaultdict
  nlp = spacy.load('de')
  
  def calculate_the_word_types(data):
      nouns = defaultdict(lambda: 0)
      verbs = defaultdict(lambda: 0)
      adjectives = defaultdict(lambda: 0)
  
      # count all tokens, but not the punctuations
      for i, row in data.iterrows():
          doc = nlp(row["name"] + " " + row["text"])
      data.set_value(i, "nr_token", len(list(map(lambda x: x.text, 
                                       filter(lambda x: x.pos_ != 'PUNCT', doc)))))
  
      # count only the adjectives
      for a in map(lambda x: x.lemma_, filter(lambda x: x.pos_ == 'ADJ', doc)):
          adjectives[a] += 1
      data.set_value(i, "nr_adj", len(list(map(lambda x: x.text, 
                                       filter(lambda x: x.pos_ == 'ADJ', doc)))))  
  
      # count only the nouns
      for n in map(lambda x: x.lemma_, filter(lambda x: x.pos_ == 'NOUN', doc)):
          nouns[n] +=1
      data.set_value(i, "nr_noun", len(list(map(lambda x: x.text, 
                                       filter(lambda x: x.pos_ == 'NOUN', doc)))))
  
      # count only the verbs
      for v in map(lambda x: x.lemma_, filter(lambda x: (x.pos_ == 'AUX') | (x.pos_ == 'VERB'), doc)):
          verbs[v] += 1
      data.set_value(i, "nr_verb", len(list(map(lambda x: x.text, 
                                       filter(lambda x: (x.pos_ == 'AUX') | (x.pos_ == 'VERB'), doc)))))  
  
      return data
  </code></pre>
  
  <p><strong>the output</strong></p>
  
  <pre><code>&gt;&gt;&gt; data
             name style      text     year       year_dt  nr_token  br_adj   nr_noun   nr_verb
  number  
  0001    Deme...     D   Deme...     1805    1805-01-01       NaN     NaN       NaN       NaN
  0002    Der ...     D   Der ...     1790    1790-01-01       NaN     NaN       NaN       NaN
  0003    Die ...     D   Die ...     1803    1803-01-01    7127.0   584.0    1328.0    1286.0
  </code></pre>
  
  <p>i think this worked back then, but not now. because my function output is the following and through testing i know, that it works, but the numbers are always in the last line only, so it overwrites itself, i think.</p>
  
  <p>where is the failiure? every tipps or suggestions to improve this function or maybe some easier solution are welcome!</p>