title: NLP - Speed of Named Entity Recognition (StanfordNER)
body: <p>I'm doing Natural Language Processing (NLP) in Python 3 and more specifically Named Entity Recognition (NER) on the Harry Potter set of books. I'm using StanfordNER, which works pretty well but takes incredible amounts of time...</p>
  
  <p>I have done some research online on why it would be this slow but I can't seem to find anything that truly suits my code, and I honestly think the problem lays more in the (bad) way I have written the code.</p>
  
  <p>So here's what I wrote for now :</p>
  
  <pre><code>import string
  from nltk.tokenize import sent_tokenize, word_tokenize
  import nltk.tag.stanford as st
  
  tagger = st.StanfordNERTagger('_path_/stanford-ner-2017-06-09/classifiers/english.all.3class.distsim.crf.ser.gz', '_path_/stanford-ner-2017-06-09/stanford-ner.jar')
  
  #this is just to read the file
  
  hp = open("books/hp1.txt", 'r', encoding='utf8')
  lhp = hp.readlines()
  
  #a small function I wrote to divide the book in sentences
  
  def get_sentences(lbook):
      sentences = []
      for k in lbook:
          j = sent_tokenize(k)
          for i in j:
              if bool(i):
                  sentences.append(i)
      return sentences
  
  #a function to divide a sentence into words
  
  def get_words(sentence):
      words = word_tokenize(sentence)
      return words
  
  sentences = get_sentences(lhp)
  
  #and now the code I wrote to get all the words labeled as PERSON by the StanfordNER tagger
  
  characters = []
      for i in sentence:
      characters = [tag[0] for tag in tagger.tag(get_words(sentences[i])) if tag[1]=="PERSON"]
      print(characters)
  </code></pre>
  
  <p>Now the problem, as I explained, is that the code takes huge amounts of time... So I'm wondering, is it normal or <strong>can I save time by rewriting the code in a better way ?</strong> If so, could you help me out ?</p>