title: Ignore UnicodeEncodeError when saving utf8 file
body: <p>I have a large string of a novel that I downloaded from Project Gutenberg. I am trying to save it to my computer, but I'm getting a UnicodeEncodeError and I don't know how to fix or ignore it.</p>
  
  <pre><code>from urllib import request
  
  # Get the text
  response = request.urlopen('http://www.gutenberg.org/files/2701/2701-0.txt')
  # Decode it using utf8
  raw = response.read().decode('utf8')
  # Save the file
  file = open('corpora/canon_texts/' + 'test', 'w')
  file.write(raw)
  file.close()
  </code></pre>
  
  <p>This gives me the following error:</p>
  
  <pre><code>UnicodeEncodeError: 'charmap' codec can't encode character '\ufeff' in position 0: character maps to &lt;undefined&gt;
  </code></pre>
  
  <p>First, I tried to remove the BOM at the beginning of the file:</p>
  
  <pre><code># We have to get rid of the pesky Byte Order Mark before we save it
  raw = raw.replace(u'\ufeff', '')
  </code></pre>
  
  <p>but I get the same error, just with a different position number:</p>
  
  <pre><code>UnicodeEncodeError: 'charmap' codec can't encode characters in position 7863-7864: character maps to &lt;undefined&gt;
  </code></pre>
  
  <p>If I look in that area I can't find the offending characters, so I don't know what to remove:</p>
  
  <pre><code>raw[7850:7900]
  </code></pre>
  
  <p>just prints out:</p>
  
  <pre><code>'   BALLENA,             Spanish.\r\n     PEKEE-NUEE-'
  </code></pre>
  
  <p>which doesn't look like it would be a problem.</p>
  
  <p>So then I tried to skip the bad lines with a try statement:</p>
  
  <pre><code>file = open('corpora/canon_texts/' + 'test', 'w')
  try:
      file.write(raw)
  except UnicodeEncodeError:
      pass
  file.close()
  </code></pre>
  
  <p>but this skips the entire text, giving me a file of 0 size.</p>
  
  <p>How can I fix this?</p>
  
  <p>EDIT:</p>
  
  <p>A couple people have noted that '\ufeff' is utf16. I tried switching to utf16:</p>
  
  <pre><code># Get the text
  response = request.urlopen('http://www.gutenberg.org/files/2701/2701-0.txt')
  # Decode it using utf16
  raw = response.read().decode('utf-16')
  </code></pre>
  
  <p>But I can't even download the data before I get this error:</p>
  
  <pre><code>UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x0a in position 1276798: truncated data
  </code></pre>
  
  <p>SECOND EDIT:</p>
  
  <p>I also tried decoding with utf-8-sig as suggested in <a href="https://stackoverflow.com/questions/17912307/u-ufeff-in-python-string">u&#39;\ufeff&#39; in Python string</a> because that includes BOM, but then I'm back to this error when I try to save it:</p>
  
  <pre><code>UnicodeEncodeError: 'charmap' codec can't encode characters in position 7863-7864: character maps to &lt;undefined&gt;
  </code></pre>