title: Scraping multiple webpages at once with Selenium
body: <p>I am using <code>selenium</code> and Python to do a big project. I have to go through 320.000 webpages (<strong>320K</strong>) one by one and scrape details and then sleep for a second and move on.</p>
  <p>Like bellow:</p>
  <pre><code>links = [&quot;https://www.thissite.com/page=1&quot;,&quot;https://www.thissite.com/page=2&quot;, &quot;https://www.thissite.com/page=3&quot;]
  
  for link in links:
      browser.get(link )
      scrapedinfo = browser.find_elements_by_xpath(&quot;*//div/productprice&quot;).text
      open(&quot;file.csv&quot;,&quot;a+&quot;).write(scrapedinfo)
      time.sleep(1)
  </code></pre>
  <h3>The greatest problem : too slow!</h3>
  <p>With this script <em>I <strong>will take days</strong> or maybe <strong>weeks</strong></em>.</p>
  <ul>
  <li>Is there a way to <em>increase speed</em>? Such as, by visiting multiple
  links at the same time and scraping all at once?</li>
  </ul>
  <p>I have spent hours finding answers on google and Stackoverflow and only found about <code>multiprocessing</code>.</p>
  <p>But, I am unable to apply it in my script.</p>