title: Web scraping: loop through result pages AND through table rows
body: <p>I appreciate all the questions and answers out there regarding Python/beautifulSoup/scraping, but I haven't seen much about this scenario and I'm stuck. Currently, my code can successfully loop through pages of a search result and create a csv doc, but when it comes to each individual table, it will only copy the first row before moving onto the next result page.</p>
  
  <p>For example, <a href="http://www.elections.ca/WPAPPS/WPR/EN/NC/Details?province=-1&amp;distyear=2013&amp;district=-1&amp;party=-1&amp;selectedid=8561" rel="nofollow noreferrer">this page</a>. Currently, my output looks like this:</p>
  
  <pre><code>Brian Benoit,25-Jun-16,Conservative,12-May-16,25-Jun-16,Medicine Hat--Cardston--Warner,b'Medicine Hat--Cardston--Warner',Nikolai Punko
  </code></pre>
  
  <p>It should look like this instead:</p>
  
  <pre><code>Brian Benoit,25-Jun-16,Conservative,12-May-16,25-Jun-16,Medicine Hat--Cardston--Warner,b'Medicine Hat--Cardston--Warner',Nikolai Punko
  Paul Hinman,25-Jun-16,Conservative,12-May-16,25-Jun-16,Medicine Hat--Cardston--Warner,b'Welling, Alberta',Robert B. Barfuss
  Michael Jones,25-Jun-16,Conservative,12-May-16,25-Jun-16,Medicine Hat--Cardston--Warner,b'Raymond, Alberta',Dawn M. Hamon 
  </code></pre>
  
  <p>(And so on for all of the rows in the table.)</p>
  
  <p>My question is: how do I get it to loop through and scrape <strong>each</strong> row before continuing to the next results page? Thanks. </p>
  
  <p>Here is my code:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import requests
  import re
  import csv
  
  
  url = "http://www.elections.ca/WPAPPS/WPR/EN/NC?province=-1&amp;distyear=2013&amp;district=-1&amp;party=-1&amp;pageno={}&amp;totalpages=55&amp;totalcount=1368&amp;secondaryaction=prev25"
  
  with open('scrapeAllRows.csv', 'w', newline='') as f_output:
      csv_output = csv.writer(f_output)
  
      for i in range(1, 56):
          print(i)
          r  = requests.get(url.format(i))
          data = r.text
          soup = BeautifulSoup(data, "html.parser")
          links = []
  
          for link in soup.find_all('a', href=re.compile('selectedid=')):
              links.append("http://www.elections.ca" + link.get('href'))
  
          for link in links:
              r  = requests.get(link)
              data = r.text
              cat = BeautifulSoup(data, "html.parser")
              header = cat.find_all('span')
              tables = cat.find_all("table")[0].find_all("td")        
  
              row = [
                  #"name": 
                  re.sub("[\n\r/]", "", cat.find_all("table")[0].find_all("td", headers="name/1")[0].contents[0]).strip(),
                  #"date": 
                  header[2].contents[0],
                  #"party": 
                  re.sub("[\n\r/]", "", cat.find("legend").contents[2]).strip(),
                  #"start_date": 
                  header[3].contents[0],
                  #"end_date": 
                  header[5].contents[0],
                  #"electoral district": 
                  re.sub("[\n\r/]", "", cat.find_all('div', class_="group")[2].contents[2]).strip(),
                  #"registered association": 
                  re.sub("[\n\r/]", "", cat.find_all('div', class_="group")[2].contents[2]).strip().encode('latin-1'),
                  #"elected": 
                  re.sub("[\n\r/]", "", cat.find_all("table")[0].find_all("td", headers="elected/1")[0].contents[0]).strip(),
                  #"address": 
                  re.sub("[\n\r/]", "", cat.find_all("table")[0].find_all("td", headers="address/1")[0].contents[0]).strip(),
                  #"financial_agent": 
                  re.sub("[\n\r/]", "", cat.find_all("table")[0].find_all("td", headers="fa/1")[0].contents[0]).strip()]
  
  
              csv_output.writerow(row)
  </code></pre>