title: Prevent BeautifulSoup's renderContents() from changing &nbsp; to Â
body: <p>I'm using <a href="https://beautiful-soup-4.readthedocs.io/en/latest/index.html" rel="nofollow noreferrer">bs4</a> to do some work on some text, but in some cases it converts <code>&amp;nbsp;</code> characters to <code>Â</code>.  The best I can tell is that this is an <a href="https://stackoverflow.com/questions/1461907/html-encoding-issues-%C3%82-character-showing-up-instead-of-nbsp">encoding mismatch from UTF-8 to latin1</a> (or reverse?)</p>
  
  <p>Everything in my web app is UTF-8, Python3 is UTF-8, and I've confirmed the database is UTF-8.</p>
  
  <p>I've narrowed down the problem to this one line:        </p>
  
  <pre><code>print("Before soup: " + text)  # Before soup: &amp;nbsp;
  soup = BeautifulSoup(text, "html.parser")
  #.... do stuff to soup, but all commented out for this testing.
  soup = BeautifulSoup(soup.renderContents(), "html.parser")  # &lt;---- PROBLEM!
  print(soup.renderContents())  # b'\xc3\x82\xc2\xa0'
  print("After SOUP: " + str(soup))  # After SOUP: Â
  </code></pre>
  
  <p>How do I prevent renderContents() from changing the encoding?  There is <a href="https://beautiful-soup-4.readthedocs.io/en/latest/search.html?q=RenderContents&amp;check_keywords=yes&amp;area=default" rel="nofollow noreferrer">no documentation</a> on this function!</p>
  
  <p>Edit: Upon further research into the docs, <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#output-formatters" rel="nofollow noreferrer">this seems to be the key</a>, but I still can't fix the problem!</p>
  
  <pre><code>print(soup.prettify(formatter="html"))  # &amp;Acirc;&amp;nbsp;
  </code></pre>