title: Python Character Encoding?
body: <p>I'm not sure with the title I'm used, but basically I want to get rid off weird characters from the string. Here is the code:</p>
  
  <pre><code>#!/usr/bin/python
  # -*- coding: utf-8 -*-
  # source: http://snippets.dzone.com/posts/show/4569
  
  from htmlentitydefs import name2codepoint as n2cp
  import re
  
  def substitute_entity(match):
      ent = match.group(3)
  
      if match.group(1) == "#":
          if match.group(2) == '':
              return unichr(int(ent))
          elif match.group(2) == 'x':
              return unichr(int('0x'+ent, 16))
      else:
          cp = n2cp.get(ent)
  
          if cp:
             return unichr(cp)
          else:
             return match.group()
  
  def decode_htmlentities(string):
      entity_re = re.compile(r'&amp;(#?)(x?)(\d{1,5}|\w{1,8});')
      return entity_re.subn(substitute_entity, string)[0]
  
  
  
  test = ['&lt;b&gt;Blogger&lt;/b&gt; in the Classroom - &lt;b&gt;Google&lt;/b&gt;', 'Of\xef\xac\x81cial &lt;b&gt;Google   Blog&lt;/b&gt;']
  container = []
  
  for i in test:
      container.append(decode_htmlentities(i))
  print container
  
  for i in test:
      print decode_htmlentities(i)
  </code></pre>
  
  <p>And here is the result:</p>
  
  <pre><code>['&lt;b&gt;Blogger&lt;/b&gt; in the Classroom - &lt;b&gt;Google&lt;/b&gt;', 'Of\xef\xac\x81cial &lt;b&gt;Google Blog&lt;/b&gt;']
  
  &lt;b&gt;Blogger&lt;/b&gt; in the Classroom - &lt;b&gt;Google&lt;/b&gt;
  Ofﬁcial &lt;b&gt;Google Blog&lt;/b&gt;
  </code></pre>
  
  <p>The Question:</p>
  
  <p>Using the same function (decode_htmlentities()), why I get a different result when appending into a list and 'just' printing?</p>
  
  <p>Here is the difference:</p>
  
  <pre><code>Of\xef\xac\x81cial &lt;b&gt;Google Blog&lt;/b&gt; # output from list
  Ofﬁcial &lt;b&gt;Google Blog&lt;/b&gt; # output from print 
  </code></pre>