title: Using BeautifulSoup to extract a table in Python 3
body: <p>I would like to use BeautifulSoup to extract a table from a website and store it as structured data. <strong>The final output I require is something that can be exported to a .csv with a header row and multiple data rows.</strong></p>
  <p>I followed the answer to <a href="https://stackoverflow.com/questions/11790535/extracting-data-from-html-table">this question</a>, but it appears updates to Python (or BeautifulSoup) require adjustments since it was posted 8 years ago. I think I have that mostly solved (see below), but in addition, the original answer seems to stop just short of actually structuring the data, instead outputting a list of header-data pairs.</p>
  <p>I'd like to use a similar solution because it seems really close to what I need. My data is already parsed using BeautifulSoup so I'm specifically asking for a solution using that package rather than Pandas.</p>
  <h1>Reproducible Example</h1>
  <p>Altered from original question by adding a second row, as my data has many rows.</p>
  <pre class="lang-py prettyprint-override"><code>from bs4 import BeautifulSoup
  
  html = &quot;&quot;&quot;
    &lt;table class=&quot;details&quot; border=&quot;0&quot; cellpadding=&quot;5&quot; cellspacing=&quot;2&quot; width=&quot;95%&quot;&gt;
      &lt;tr valign=&quot;top&quot;&gt;
        &lt;th&gt;Tests&lt;/th&gt;
        &lt;th&gt;Failures&lt;/th&gt;
        &lt;th&gt;Success Rate&lt;/th&gt;
        &lt;th&gt;Average Time&lt;/th&gt;
        &lt;th&gt;Min Time&lt;/th&gt;
        &lt;th&gt;Max Time&lt;/th&gt;
     &lt;/tr&gt;
     &lt;tr valign=&quot;top&quot; class=&quot;Failure&quot;&gt;
       &lt;td&gt;103&lt;/td&gt;
       &lt;td&gt;24&lt;/td&gt;
       &lt;td&gt;76.70%&lt;/td&gt;
       &lt;td&gt;71 ms&lt;/td&gt;
       &lt;td&gt;0 ms&lt;/td&gt;
       &lt;td&gt;829 ms&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr valign=&quot;top&quot; class=&quot;Failure&quot;&gt;
       &lt;td&gt;109&lt;/td&gt;
       &lt;td&gt;35&lt;/td&gt;
       &lt;td&gt;82.01%&lt;/td&gt;
       &lt;td&gt;12 ms&lt;/td&gt;
       &lt;td&gt;2 ms&lt;/td&gt;
       &lt;td&gt;923 ms&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/table&gt;&quot;&quot;&quot;
  
  soup = BeautifulSoup(html)
  table = soup.find(&quot;table&quot;, attrs={&quot;class&quot;:&quot;details&quot;})
  
  # The first tr contains the field names.
  headings = [th.get_text() for th in table.find(&quot;tr&quot;).find_all(&quot;th&quot;)]
  
  datasets = []
  for row in table.find_all(&quot;tr&quot;)[1:]:
      dataset = zip(headings, (td.get_text() for td in row.find_all(&quot;td&quot;)))
      datasets.append(dataset)
  
  print(datasets)
  </code></pre>
  <p>The result is supposed to look like the following (though with multiple rows, I'm not sure precisely the structure).</p>
  <pre class="lang-py prettyprint-override"><code>[[(u'Tests', u'103'),
    (u'Failures', u'24'),
    (u'Success Rate', u'76.70%'),
    (u'Average Time', u'71 ms'),
    (u'Min Time', u'0 ms'),
    (u'Max Time', u'829 ms')]]
  </code></pre>
  <p>But instead looks like:</p>
  <pre class="lang-py prettyprint-override"><code>[&lt;zip object at 0x7fb06b5efdc0&gt;, &lt;zip object at 0x7fb06b5ef980&gt;]
  </code></pre>
  <h1>Attempted Solution</h1>
  <p>I tried using <code>datasets.append(tuple(dataset))</code> in the existing for loop, which resulted in:</p>
  <pre class="lang-py prettyprint-override"><code>[(('Tests', '103'), ('Failures', '24'), ('Success Rate', '76.70%'), ('Average Time', '71 ms'), ('Min Time', '0 ms'), ('Max Time', '829 ms')), 
  (('Tests', '109'), ('Failures', '35'), ('Success Rate', '82.01%'), ('Average Time', '12 ms'), ('Min Time', '2 ms'), ('Max Time', '923 ms'))]
  </code></pre>
  <p>This is closer to the original answer's expected output, but obviously duplicates the pairs rather than creating a data table with headers and values. So I'm not sure what to do with the data from this point.</p>