title: Scraping a table with row labels in Python using Beautiful Soup
body: <p>I'm trying to scrape a table from a website that has row labels.  I'm able to get the actual data from the table, but I have no idea how to get the row labels as well.</p>
  
  <p>Here is my code right now:</p>
  
  <pre><code>import numpy as np
  import pandas as pd  
  import urllib.request
  from bs4 import BeautifulSoup
  
  url = "http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/dt-td/Rp-eng.cfm?TABID=2&amp;LANG=E&amp;A=R&amp;APATH=3&amp;DETAIL=0&amp;DIM=0&amp;FL=A&amp;FREE=0&amp;GC=01&amp;GL=-1&amp;GID=1341679&amp;GK=1&amp;GRP=1&amp;O=D&amp;PID=110719&amp;PRID=10&amp;PTYPE=109445&amp;S=0&amp;SHOWALL=0&amp;SUB=0&amp;Temporal=2017&amp;THEME=125&amp;VID=0&amp;VNAMEE=&amp;VNAMEF=&amp;D1=0&amp;D2=0&amp;D3=0&amp;D4=0&amp;D5=0&amp;D6=0"
  res = urllib.request.urlopen(url)
  
  html = res.read()
  
  ## parse with BeautifulSoup
  bs = BeautifulSoup(html, "html.parser")
  
  tables = bs.find_all("table")
  table = tables[0]
  
  df = pd.DataFrame()
  
  rows = table.find_all("tr")
  
  #extract the first column name (Employment income groups (18))
  column_names = []
  header_cells = rows[0].find_all("th") 
  
  for cell in header_cells:
      header = cell.text
      header = header.strip()
      header = header.replace("\n", " ")
      column_names.append(header)
  
  #extract the rest of the column names
  header_cells = rows[1].find_all("th") 
  
  for cell in header_cells:
      header = cell.text
      header = header.strip()
      header = header.replace("\n", " ")
      column_names.append(header)
  
  #this is an extra label
  column_names.remove('Main mode of commuting (10)')
  
  #get the data from the table
  data = []
  for row in rows[2:]:
  
      ## create an empty tuple
      dt = ()
  
      cells = row.find_all("td")
  
      for cell in cells:
          ## dp stands for "data point"
          font = cell.find("font")
  
          if font is not None:
              dp = font.text
          else:
              dp = cell.text
  
          dp = dp.strip()
          dp = dp.replace("\n", " ")
  
          ## add to tuple
          dt = dt + (dp,)
      data.append(dt)
  
  df = pd.DataFrame(data, columns = column_names)
  </code></pre>
  
  <p>Creating the dataframe will give an error because the code above only extracts the cells with data points but does not extract the first cell of each row that contains the row label.</p>
  
  <p>That is, there are 11 column names, but the tuples only have 10 values because it is not extracting the row label (ie, Total - Employment income) because they are of "th" type.</p>
  
  <p>How can I get the row label and put it into the tuple as I process the rest of the data in the table?</p>
  
  <p>Thank you for your help.</p>
  
  <p>(The table I am trying to scrape is on this <a href="http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/dt-td/Rp-eng.cfm?TABID=2&amp;LANG=E&amp;A=R&amp;APATH=3&amp;DETAIL=0&amp;DIM=0&amp;FL=A&amp;FREE=0&amp;GC=01&amp;GL=-1&amp;GID=1341679&amp;GK=1&amp;GRP=1&amp;O=D&amp;PID=110719&amp;PRID=10&amp;PTYPE=109445&amp;S=0&amp;SHOWALL=0&amp;SUB=0&amp;Temporal=2017&amp;THEME=125&amp;VID=0&amp;VNAMEE=&amp;VNAMEF=&amp;D1=0&amp;D2=0&amp;D3=0&amp;D4=0&amp;D5=0&amp;D6=0" rel="nofollow noreferrer">site</a> if it's not clear from the code)</p>