title: Faster way for scraping over 100,000 web pages in Python than Selenium?
body: <p>I have over 100,000 web pages to scrape data from. The pages do not seem to have any dynamic data we need to wait to load (or that I need) so I am assuming there is no need to execute Javascript. I am currently using Selenium to do the job for me but it is currently using the Chrome Driver to do the scraping:</p>
  
  <pre><code>chromeOptions = webdriver.ChromeOptions()
  prefs = {"profile.managed_default_content_settings.images":2}
  chromeOptions.add_experimental_option("prefs",prefs)
  
  driver = webdriver.Chrome(chrome_options=chromeOptions)
  </code></pre>
  
  <p>I have attempted to use the PhantomJS driver under the logic that Headless approach would be faster but it seems to be running much slower (an extra 2 seconds per page to load compared to the Chrome Driver):</p>
  
  <pre><code>webdriver.DesiredCapabilities.PHANTOMJS['javascriptEnabled'] = False
  driver = webdriver.PhantomJS(service_args=['--load-images=no'])
  driver.set_window_size(1120, 550)
  </code></pre>
  
  <p>Am I missing something obvious with regards to the PHantomJS driver? My approach for scraping the data uses a mixture of CSS selectors and finding elements by ID and name to scrape the data I need.  Also, it would be in my best interest to do a multi-threaded approach here as well to cut back on waiting for this to finish. Selenium does not seem to do multi-threading well from what I understand. Does another module allow for this?</p>