title: Unable to read non ascii characters from csv file
body: <p>I am trying to read csv file which contains a sentence each line. Each sentence may contain foreign words such as Chinese characters. I want to remove or ignore those foreign characters and return only English words or ascii words.</p>
  
  <p>Example of how the string may look like in the file:</p>
  
  <pre><code>'小心 Careful'
  </code></pre>
  
  <p>Desired output: Careful</p>
  
  <pre><code>import csv
  from string import ascii_letters, punctuation
  
  def remove_non_ascii(string):
      ascii_letters = set(ascii_letters)
      tokens = nltk.word_tokenize(string)
      ascii_words = [word for word in tokens if any(letter in ascii_letters for letter in word)]
      return ascii_words
  
  with open(job_file, mode = 'r', encoding = 'utf8') as infile:   
      line_reader  = csv.reader(infile)
      for row in line_reader:
          new_line = remove_non_ascii(row[1])
          print (new_line)
          if row[1]:
              open(output_file, 'a', newline='', encoding = 'utf8') as outfile:
              line_writer = csv.writer(outfile)
              line_writer.writerow('')             
  </code></pre>
  
  <p>This is the error I get when I run that code.</p>
  
  <pre><code>UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 2848: invalid start byte
  </code></pre>
  
  <p>The error goes away if I change the encoding from utf8 to cp1252 because the Chinese characters are converted into '????'. Is it possible to remove those unwanted characters and return only ascii compliant characters?</p>