title: How to concatenate two html file bodies with BeautifulSoup?
body: <p><strong>I need to concatenate the bodies of two html files into one html file, with a bit of arbitrary html as a separator in between.  I have code that used to work for this, but stopped working</strong> when I upgraded from Xubuntu 11.10 (or was it 11.04?) to 12.10, probably due to a BeautifulSoup update (I'm currently using 3.2.1; I don't know what version I had previously) or to a vim update (I use vim to auto-generate the html files from plaintext ones). This is the stripped-down version of the code:</p>
  
  <pre><code>from BeautifulSoup import BeautifulSoup
  soup_original_1 = BeautifulSoup(''.join(open('test1.html')))
  soup_original_2 = BeautifulSoup(''.join(open('test2.html')))
  contents_1 = soup_original_1.body.renderContents()
  contents_2 = soup_original_2.body.renderContents()
  contents_both = contents_1 + "\n&lt;b&gt;SEPARATOR\n&lt;/b&gt;" + contents_2
  soup_new = BeautifulSoup(''.join(open('test1.html')))
  while len(soup_new.body.contents):
      soup_new.body.contents[0].extract()
  soup_new.body.insert(0, contents_both)                       
  </code></pre>
  
  <p>The bodies of the two input files used for the test case are very simple: <code>contents_1</code> is <code>\n&lt;pre&gt;\nFile 1\n&lt;/pre&gt;\n'</code> and <code>contents_2</code> is <code>'\n&lt;pre&gt;\nFile 2\n&lt;/pre&gt;\n'</code>. </p>
  
  <p>I would like <code>soup_new.body.renderContents()</code> to be a concatenation of those two with the separator text in between, but instead all the <code>&lt;</code>'s change into <code>&amp;lt;</code> etc. - the desired result is <code>'\n&lt;pre&gt;\nFile 1\n&lt;/pre&gt;\n\n&lt;b&gt;SEPARATOR\n&lt;/b&gt;\n&lt;pre&gt;\nFile 2\n&lt;/pre&gt;\n'</code>, which is what I used to get prior to the OS update; the current result is <code>'\n&amp;lt;pre&amp;gt;\nFile 1\n&amp;lt;/pre&amp;gt;\n\n&amp;lt;b&amp;gt;SEPARATOR\n&amp;lt;/b&amp;gt;\n&amp;lt;pre&amp;gt;\nFile 2\n&amp;lt;/pre&amp;gt;\n'</code>, which is pretty useless.</p>
  
  <p><strong>How do I make BeautifulSoup stop turning <code>&lt;</code> into <code>&amp;lt;</code> etc when inserting html as a string into a soup object's body?  Or should I just be doing this in an entirely different way?</strong> (This is my only experience with BeautifulSoup and most other html parsing, so I'm guessing this may well be the case.)</p>
  
  <p>The html files are automatically generated from plaintext files with vim (the real cases I use are obviously more complicated, and involve custom syntax highlighting, which is why I'm doing it this way at all).  The full test1.html file looks like this, and test2.html is identical except for contents and title.</p>
  
  <pre><code>&lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"&gt;
  &lt;html&gt;
  &lt;head&gt;
  &lt;meta http-equiv="content-type" content="text/html; charset=utf-8" /&gt;
  &lt;title&gt;~/programs/lab_notebook_and_printing/concatenate-html_problem_2013/test1.txt.html&lt;/title&gt;
  &lt;meta name="Generator" content="Vim/7.3" /&gt;
  &lt;meta name="plugin-version" content="vim7.3_v10" /&gt;
  &lt;meta name="syntax" content="none" /&gt;
  &lt;meta name="settings" content="ignore_folding,use_css,pre_wrap,expand_tabs,ignore_conceal" /&gt;
  &lt;style type="text/css"&gt;
  pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; white-space: pre-wrap; word-wrap: break-word }
  body { font-family: monospace; color: #000000; background-color: #ffffff; font-size: 0.875em }
  &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
  &lt;pre&gt;
  File 1
  &lt;/pre&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>