title: Scraping wunderground without API, using python
body: <p>I'm not very experienced in the world of scraping data, so the problem here may be obvious to some.</p>
  
  <p>What I want is to scrape historical daily weather data from wunderground.com, without paying the API. Maybe it's not possible at all.</p>
  
  <p>My method is simply to use <code>requests.get</code> and save the whole text into a file (code below). </p>
  
  <p>Instead of getting the tables that can be accessed from the web browser (see image below), the result is a file that has almost everything but those tables. Something like this:</p>
  
  <p><strong>Summary</strong><br>
  No data recorded<br>
  <strong>Daily Observations</strong><br>
  No Data Recorded  </p>
  
  <p>What is weird is that if I save-as the web page with Firefox, the result depends on whether I choose 'web-page, only HTML' or 'web-page, complete': the latter includes the data I'm interested in, the former does not.</p>
  
  <p>Is it possible that this is on purpose so nobody scrapes their data? I just wanted to make sure there is not a workaround this problem.</p>
  
  <p>Thanks in advance,
  Juan</p>
  
  <p>Note: I tried using the <a href="https://en.wikipedia.org/wiki/User_agent#User_agent_identification" rel="noreferrer">user-agent</a> field to no avail.</p>
  
  <pre class="lang-py prettyprint-override"><code># Note: I run &gt; set PYTHONIOENCODING=utf-8 before executing python
  import requests
  
  # URL with wunderground weather information for a specific date:
  date = '2019-03-12'
  url = 'https://www.wunderground.com/history/daily/sd/khartoum/HSSS/date/' + date
  r = requests.get(url)
  
  # Write a file to check if the tables ar being retrieved:
  with open('test.html', 'wb') as testfile:
      testfile.write(r.text.encode('utf-8'))
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/Y6x7z.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Y6x7z.png" alt="Screenshot of the tables I want to scrape."></a></p>
  
  <hr>
  
  <p><strong>UPDATE: FOUND A SOLUTION</strong> </p>
  
  <p>Thanks to pointing me to the selenium module, it is the exact solution I needed. The code extracts all the tables present on the URL of a given date (as seen when visiting the site normally). It needs modifications in order to be able to scrape over a list of dates and organize the CSV files created.</p>
  
  <p><strong>Note</strong>: <a href="https://github.com/mozilla/geckodriver/releases" rel="noreferrer">geckodriver.exe</a> is needed in the working directory. </p>
  
  <pre><code>from bs4 import BeautifulSoup
  from selenium import webdriver
  from selenium.webdriver.common.action_chains import ActionChains
  from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
  from selenium.webdriver.common.keys import Keys
  import requests, sys, re
  
  # URL with wunderground weather information
  url = 'https://www.wunderground.com/history/daily/sd/khartoum/HSSS/date/2019-3-12'
  
  # Commands related to the webdriver (not sure what they do, but I can guess):
  bi = FirefoxBinary(r'C:\Program Files (x86)\Mozilla Firefox\\firefox.exe')
  br = webdriver.Firefox(firefox_binary=bi)
  
  # This starts an instance of Firefox at the specified URL:
  br.get(url)
  
  # I understand that at this point the data is in html format and can be
  # extracted with BeautifulSoup:
  sopa = BeautifulSoup(br.page_source, 'lxml')
  
  # Close the firefox instance started before:
  br.quit()
  
  # I'm only interested in the tables contained on the page:
  tablas = sopa.find_all('table')
  
  # Write all the tables into csv files:
  for i in range(len(tablas)):
      out_file = open('wunderground' + str(i + 1) + '.csv', 'w')
      tabla = tablas[i]
  
      # ---- Write the table header: ----
      table_head = tabla.findAll('th')
      output_head = []
      for head in table_head:
          output_head.append(head.text.strip())
  
      # Some cleaning and formatting of the text before writing:
      encabezado = '"' + '";"'.join(output_head) + '"'
      encabezado = re.sub('\s', '', encabezado) + '\n'
      out_file.write(encabezado.encode(encoding='UTF-8'))
  
      # ---- Write the rows: ----
      output_rows = []
      filas = tabla.findAll('tr')
      for j in range(1, len(filas)):
          table_row = filas[j]
          columns = table_row.findAll('td')
          output_row = []
          for column in columns:
              output_row.append(column.text.strip())
  
          # Some cleaning and formatting of the text before writing:
          fila = '"' + '";"'.join(output_row) + '"'
          fila = re.sub('\s', '', fila) + '\n'
          out_file.write(fila.encode(encoding='UTF-8'))
  
      out_file.close()
  
  </code></pre>
  
  <p><strong>Extra</strong>: the answer of @QHarr works beautifully, but I needed a couple of modifications to use it, because I use firefox in my PC. It's important to note that for this to work I had to add the <a href="https://github.com/mozilla/geckodriver/releases" rel="noreferrer">geckodriver.exe</a> file into my working directory. Here's the code:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
  from selenium.webdriver.support import expected_conditions as EC
  import pandas as pd
  
  url = 'https://www.wunderground.com/history/daily/sd/khartoum/HSSS/date/2019-03-12'
  bi = FirefoxBinary(r'C:\Program Files (x86)\Mozilla Firefox\\firefox.exe')
  driver = webdriver.Firefox(firefox_binary=bi)
  # driver = webdriver.Chrome()
  driver.get(url)
  tables = WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table")))
  for table in tables:
      newTable = pd.read_html(table.get_attribute('outerHTML'))
      if newTable:
          print(newTable[0].fillna(''))
  </code></pre>