title: List index out of range with stanford-nlp
body: <p>I'm trying to remove all blank lines from a large .txt file but whatever method I use it always returns this traceback:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;C:\Users\svp12\PycharmProjects\practiques\main.py&quot;, line 53, in &lt;module&gt;
      doc = nlp(texts[line])
  IndexError: list index out of range
  </code></pre>
  <p>If I don't remove these spaces then I get IndexErrors on the consequent 2 for loops (or at least I think that's the reason), that's why I'm using the the try/except like this:</p>
  <pre><code>try:
      for word in doc.sentences[0].words:
          noun.append(word.text)
          lemma.append(word.lemma)
          pos.append(word.pos)
          xpos.append(word.xpos)
          deprel.append(word.deprel)
  except IndexError:
      errors += 1
      pass
  </code></pre>
  <p>I'd like to be able to remove all blank lines and not have to avoid IndexErrors like this, any idea on how to fix?</p>
  <p>Here's the whole code:</p>
  <pre><code>import io
  import stanza
  import os
  
  
  def linecount(filename):
      ffile = open(filename, 'rb')
      lines = 0
      buf_size = 1024 * 1024
      read_f = ffile.read
  
      buf = read_f(buf_size)
      while buf:
          lines += buf.count(b'\n')
          buf = read_f(buf_size)
  
      return lines
  
  
  errors = 0
  
  with io.open('@_Calvia_2018-01-01_2022-04-01.txt', 'r+', encoding='utf-8') as f:
      text = f.read()
  
  # replacing eos with \n, numbers and symbols
  texts = text.replace('eos', '.\n')
  texts = texts.replace('0', ' ').replace('1', ' ').replace('2', ' ').replace('3', ' ').replace('4', ' ')\
      .replace('5', ' ').replace('6', ' ').replace('7', ' ').replace('8', ' ').replace('9', ' ').replace(',', ' ')\
      .replace('&quot;', ' ').replace('·', ' ').replace('?', ' ').replace('¿', ' ').replace(':', ' ').replace(';', ' ')\
      .replace('-', ' ').replace('!', ' ').replace('¡', ' ').replace('.', ' ').splitlines()
  
  os.system(&quot;sed -i \'/^$/d\' @_Calvia_2018-01-01_2022-04-01.txt&quot;)            # removing empty lines to avoid IndexError
  
  nlp = stanza.Pipeline(lang='ca')
  
  nouns = []
  lemmas = []
  poses = []
  xposes = []
  heads = []
  deprels = []
  
  total_lines = linecount('@_Calvia_2018-01-01_2022-04-01.txt') - 1
  
  for line in range(50):                                                  # range should be total_lines which is 6682
      noun = []
      lemma = []
      pos = []
      xpos = []
      head = []
      deprel = []
      # print('analyzing: '+str(line+1)+' / '+str(len(texts)), end='\r')
      doc = nlp(texts[line])
      try:
          for word in doc.sentences[0].words:
              noun.append(word.text)
              lemma.append(word.lemma)
              pos.append(word.pos)
              xpos.append(word.xpos)
              deprel.append(word.deprel)
      except IndexError:
          errors += 1
          pass
      try:
          for word in doc.sentences[0].words:
              head.extend([lemma[word.head-1] if word.head &gt; 0 else &quot;root&quot;])
      except IndexError:
          errors += 1
          pass
      nouns.append(noun)
      lemmas.append(lemma)
      poses.append(pos)
      xposes.append(xpos)
      heads.append(head)
      deprels.append(deprel)
  
  print(nouns)
  print(lemmas)
  print(poses)
  print(xposes)
  print(heads)
  print(deprels)
  
  print(&quot;errors: &quot; + str(errors))                                                         # wierd, seems to be range/2-1
  </code></pre>
  <p>And as a side question, is worth to import os just for this line? (which is the one removing the blank lines</p>
  <pre><code>os.system(&quot;sed -i \'/^$/d\' @_Calvia_2018-01-01_2022-04-01.txt&quot;)
  </code></pre>