title: How can I run web scraper script automatically every hour?
body: <p>I am extracting data from booking.com, my script uses selenium to gather the data, creates a provisional csv with the appropriate timestamp and then appends it to the final database which is also a csv. I would like to get new data every hour even when I'm offline, and append it to the final database, yet I don't know how to do it. I am new to web scraping. Currently my script runs in Jupyter. Any help would be greatly appreciated.</p>
  <p>I'm using macOS Big Sur</p>
  <p>This is my code:</p>
  <pre><code> 
  def prepare_driver(url):
      '''Returns a Firefox Webdriver.'''
      options = Options()
      # options.add_argument('-headless')
      driver = Firefox(executable_path='/Users/andreazavala/Downloads/geckodriver', options=options)
      driver.get(url)
      wait = WebDriverWait(driver, 10).until(EC.presence_of_element_located(
          (By.ID, 'ss')))
      return driver
  
  def fill_form(driver, search_argument):
      '''Finds all the input tags in form and makes a POST requests.'''
      search_field = driver.find_element_by_id('ss')
      search_field.send_keys(search_argument)
      
      #Look for today's date
      driver.find_element_by_class_name('xp__dates-inner').click()
      slcpath = &quot;td[data-date='&quot;+str(date.today())+&quot;']&quot;
      driver.find_element_by_css_selector(slcpath).click()
      
      # We look for the search button and click it
      driver.find_element_by_class_name('sb-searchbox__button')\
          .click()
      
      wait = WebDriverWait(driver, timeout=10).until(
          EC.presence_of_all_elements_located(
              (By.CLASS_NAME, 'sr-hotel__title')))
  
  driver = prepare_driver(domain)
  fill_form(driver, 'City Name')
  
  url_iter = driver.current_url
  accommodation_urls = list()
  accommodation_urls.append(url_iter)
  
  with open('urls.txt', 'w') as f:
      for item in accommodation_urls:
          f.write(&quot;%s\n&quot; % item)
  from selectorlib import Extractor
  import requests 
  from time import sleep
  import csv
  
  # Create an Extractor by reading from the YAML file
  e = Extractor.from_yaml_file('booking.yml')
  
  def scrape(url):    
      headers = {
          'Connection': 'keep-alive',
          'Pragma': 'no-cache',
          'Cache-Control': 'no-cache',
          'DNT': '1',
          'Upgrade-Insecure-Requests': '1',
          # You may want to change the user agent if you get blocked
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36',
          'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  
          'Referer': 'https://www.booking.com/index.en-gb.html',
          'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
      }
  
      # Download the page using requests
      print(&quot;Downloading %s&quot;%url)
      r = requests.get(url, headers=headers)
      # Pass the HTML of the page and create 
      return e.extract(r.text,base_url=url)
  
  
  with open(&quot;urls.txt&quot;,'r') as urllist, open('data.csv','w') as outfile:
      fieldnames = [
          &quot;name&quot;,
          &quot;location&quot;,
          &quot;price&quot;,
          &quot;price_for&quot;,
          &quot;room_type&quot;,
          &quot;beds&quot;,
          &quot;rating&quot;,
          &quot;rating_title&quot;,
          &quot;number_of_ratings&quot;,
          &quot;url&quot;
      ]
      writer = csv.DictWriter(outfile, fieldnames=fieldnames,quoting=csv.QUOTE_ALL)
      writer.writeheader()
      for url in urllist.readlines():
          data = scrape(url) 
          if data:
              for h in data['hotels']:
                  writer.writerow(h)
  import pandas as pd
  data = pd.read_csv(&quot;data.csv&quot;)
  data.insert(0, 'TimeStamp', pd.to_datetime('today').replace(microsecond=0))
  
  df2 = data
  df2.to_csv('Tarifa.csv', mode = 'a', header = False)
  df_results = pd.read_csv('Tarifa.csv', index_col=0).reset_index(drop = True, inplace = True)
  </code></pre>