title: Python: parse html and produce a tabular text file
body: <p><strong>The problem:</strong> I want to parse an html code and retrieve a file of tabular text such as this:</p>
  
  <pre><code>East Counties
  Babergh, http://ratings.food.gov.uk/OpenDataFiles/FHRS297en-GB.xml, 876
  Basildon, http://ratings.food.gov.uk/OpenDataFiles/FHRS109en-GB.xml, 1134
  ...
  ...
  </code></pre>
  
  <p><strong>What I get instead:</strong>
  Only <code>East Counties</code> appears in the txt file, so the for loop fails to print each new region. Attempt code is after the html code.</p>
  
  <p><strong>HTML code:</strong>
  The code can be found in <a href="http://view-source:http://ratings.food.gov.uk/open-data/" rel="nofollow noreferrer">this html page</a>, of which this is the excerpt referring to the above table: </p>
  
  <pre><code>&lt;h2&gt;
                                      East Counties&lt;/h2&gt;
  
                                          &lt;table&gt;
                                              &lt;thead&gt;
                                                  &lt;tr&gt;
                                                      &lt;th&gt;
                                                          &lt;span id="listRegions_lvFiles_0_titleLAName_0"&gt;Local authority&lt;/span&gt;
                                                      &lt;/th&gt;
                                                      &lt;th&gt;
                                                          &lt;span id="listRegions_lvFiles_0_titleUpdate_0"&gt;Last update&lt;/span&gt;
                                                      &lt;/th&gt;
                                                      &lt;th&gt;
                                                          &lt;span id="listRegions_lvFiles_0_titleEstablishments_0"&gt;Number of businesses&lt;/span&gt;
                                                      &lt;/th&gt;
                                                      &lt;th&gt;
                                                          &lt;span id="listRegions_lvFiles_0_titleCulture_0"&gt;Download&lt;/span&gt;
                                                      &lt;/th&gt;
                                                  &lt;/tr&gt;
                                              &lt;/thead&gt;
  
                                          &lt;tr&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_laNameLabel_0"&gt;Babergh&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_updatedLabel_0"&gt;04/05/2017 &lt;/span&gt;
                                                  at
                                                  &lt;span id="listRegions_lvFiles_0_updatedTime_0"&gt; 12:00&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_establishmentsLabel_0"&gt;876&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;a id="listRegions_lvFiles_0_fileURLLabel_0" title="Babergh: English language" href="http://ratings.food.gov.uk/OpenDataFiles/FHRS297en-GB.xml"&gt;English language&lt;/a&gt;
                                              &lt;/td&gt;
                                          &lt;/tr&gt;
  
                                          &lt;tr&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_laNameLabel_1"&gt;Basildon&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_updatedLabel_1"&gt;06/05/2017 &lt;/span&gt;
                                                  at
                                                  &lt;span id="listRegions_lvFiles_0_updatedTime_1"&gt; 12:00&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;span id="listRegions_lvFiles_0_establishmentsLabel_1"&gt;1,134&lt;/span&gt;
                                              &lt;/td&gt;
                                              &lt;td&gt;
                                                  &lt;a id="listRegions_lvFiles_0_fileURLLabel_1" title="Basildon: English language" href="http://ratings.food.gov.uk/OpenDataFiles/FHRS109en-GB.xml"&gt;English language&lt;/a&gt;
                                              &lt;/td&gt;
                                          &lt;/tr&gt;
  </code></pre>
  
  <p><strong>My attempt:</strong></p>
  
  <pre><code>from xml.dom import minidom
  import urllib2
  from bs4 import BeautifulSoup
  
  url='http://ratings.food.gov.uk/open-data/'
  f = urllib2.urlopen(url)
  mainpage = f.read()
  soup = BeautifulSoup(mainpage, 'html.parser')
  
  regions=[]
  with open('Regions_and_files.txt', 'w') as f:
      for h2 in soup.find_all('h2')[6:]: #Skip 6 h2 lines 
          region=h2.text.strip() #Get the text of each h2 without the white spaces
          regions.append(str(region))
          f.write(region+'\n')
          for tr in soup.find_all('tr')[1:]: # Skip headers
              tds = tr.find_all('td')
              if len(tds)==0:
                  continue
              else:
                  a = tr.find_all('a')
                  link = str(a)[10:67]
                  span = tr.find_all('span')
                  places = int(str(span[3].text).replace(',', ''))
                  f.write("%s,%s,%s" % \
                                (str(tds[0].text)[1:-1], link, places)+'\n')
  </code></pre>
  
  <p>How can I fix this?</p>