<p>The following implementation achieves this in <strong>16 seconds</strong>.</p>
<p>To speed up the execution process I have taken the following measures:</p>
<ul>
<li>Removed <code>Selenium</code> entirely (No clicking required)</li>
<li>For <code>abstract</code>, used <code>BeautifulSoup</code>'s output and processed it later</li>
<li>Added <code>multiprocessing</code> to speed up the process significantly</li>
</ul>
<pre class="lang-py prettyprint-override"><code>from multiprocessing import Process, Manager
import requests 
from bs4 import BeautifulSoup
import re
import time

start_time = time.time()

def get_no_of_pages(showing_text):
    no_of_results = int((re.findall(r&quot;(\d+,*\d+) results for all&quot;,showing_text)[0].replace(',','')))
    pages = no_of_results//200 + 1
    print(&quot;total pages:&quot;,pages)
    return pages 

def clean(text):
    return text.replace(&quot;\n&quot;, '').replace(&quot;  &quot;,'')

def get_data_from_page(url,page_number,data):
    print(&quot;getting page&quot;,page_number)
    response = requests.get(url+&quot;start=&quot;+str(page_number*200))
    soup = BeautifulSoup(response.content, &quot;lxml&quot;)
    
    arxiv_results = soup.find_all(&quot;li&quot;,{&quot;class&quot;,&quot;arxiv-result&quot;})

    for arxiv_result in arxiv_results:
        paper = {} 
        paper[&quot;titles&quot;]= clean(arxiv_result.find(&quot;p&quot;,{&quot;class&quot;,&quot;title is-5 mathjax&quot;}).text)
        links = arxiv_result.find_all(&quot;a&quot;)
        paper[&quot;arxiv_ids&quot;]= links[0].text.replace('arXiv:','')
        paper[&quot;arxiv_links&quot;]= links[0].get('href')
        paper[&quot;pdf_link&quot;]= links[1].get('href')
        paper[&quot;authors&quot;]= clean(arxiv_result.find(&quot;p&quot;,{&quot;class&quot;,&quot;authors&quot;}).text.replace('Authors:',''))

        split_abstract = arxiv_result.find(&quot;p&quot;,{&quot;class&quot;:&quot;abstract mathjax&quot;}).text.split(&quot;▽ More\n\n\n&quot;,1)
        if len(split_abstract) == 2:
            paper[&quot;abstract&quot;] = clean(split_abstract[1].replace(&quot;△ Less&quot;,''))
        else: 
            paper[&quot;abstract&quot;] = clean(split_abstract[0].replace(&quot;△ Less&quot;,''))

        paper[&quot;date&quot;] = re.split(r&quot;Submitted|;&quot;,arxiv_results[0].find(&quot;p&quot;,{&quot;class&quot;:&quot;is-size-7&quot;}).text)[1]
        paper[&quot;tag&quot;] = clean(arxiv_results[0].find(&quot;div&quot;,{&quot;class&quot;:&quot;tags is-inline-block&quot;}).text) 
        doi = arxiv_results[0].find(&quot;div&quot;,{&quot;class&quot;:&quot;tags has-addons&quot;})       
        if doi is None:
            paper[&quot;doi&quot;] = &quot;None&quot;
        else:
            paper[&quot;doi&quot;] = re.split(r'\s', doi.text)[1] 

        data.append(paper)
    
    print(f&quot;page {page_number} done&quot;)


if __name__ == &quot;__main__&quot;:
    url = 'https://arxiv.org/search/?searchtype=all&amp;query=healthcare&amp;abstracts=show&amp;size=200&amp;order=-announced_date_first&amp;'

    response = requests.get(url+&quot;start=0&quot;)
    soup = BeautifulSoup(response.content, &quot;lxml&quot;)

    with Manager() as manager:
        data = manager.list()  
        processes = []
        get_data_from_page(url,0,data)


        showing_text = soup.find(&quot;h1&quot;,{&quot;class&quot;:&quot;title is-clearfix&quot;}).text
        for i in range(1,get_no_of_pages(showing_text)):
            p = Process(target=get_data_from_page, args=(url,i,data))
            p.start()
            processes.append(p)

        for p in processes:
            p.join()

        print(&quot;Number of entires scraped:&quot;,len(data))

        stop_time = time.time()

        print(&quot;Time taken:&quot;, stop_time-start_time,&quot;seconds&quot;)
</code></pre>
<p>Output:</p>
<pre><code>&gt;&gt;&gt; python test.py
</code></pre>
<pre><code>getting page 0
page 0 done
total pages: 10
getting page 1
getting page 4
getting page 2
getting page 6
getting page 5
getting page 3
getting page 7
getting page 9
getting page 8
page 9 done
page 4 done
page 1 done
page 6 done
page 2 done
page 7 done
page 3 done
page 5 done
page 8 done
Number of entires scraped: 1890
Time taken: 15.911492586135864 seconds
</code></pre>
<p>Note:</p>
<ul>
<li>Please write the above code in a <code>.py</code> file. For Jupyter notebook refer <a href="https://stackoverflow.com/a/47374811/11573842">this</a>.</li>
<li>Multiprocessing code taken from <a href="https://stackoverflow.com/a/42490484/11573842">here</a>.</li>
<li>The ordering of entries in the <code>data</code> list won't match the ordering on the website as <code>Manager</code> will append <code>dictionaries</code> into it as they come.</li>
<li>The above code finds the number of pages on its own and is thus generalized to work on any arxiv search result. Unfortunately, to do this it first <code>gets</code> <code>page 0</code> and then calculates the <code>number of pages</code> and then goes for <code>multiprocessing</code> for the remaining pages. This has the disadvantage that while the <code>0th page</code> was being worked on, no other process was running. So if you remove that part and simply run the loop for <code>10 pages</code> then the time taken should fall at around <strong>8 seconds</strong>.</li>
</ul>
Is there a way to optimize the for loop? Selenium is taking too long to scrape 38 pages
<p>I'm trying to scrape <a href="https://arxiv.org/search/?query=healthcare&amp;searchtype=allI" rel="nofollow noreferrer">https://arxiv.org/search/?query=healthcare&amp;searchtype=allI</a> through the Selenium and python. The for loop takes too long to execute. I tried to scrape with headless browsers and PhantomJS, but it doesnt scrape the abstract field (Need the abstract field expanded with the more button clicked)</p>
<pre class="lang-py prettyprint-override"><code>import pandas as pd
import selenium
import re
import time
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver import Firefox

browser = Firefox()
url_healthcare = 'https://arxiv.org/search/?query=healthcare&amp;searchtype=all'
browser.get(url_healthcare)

dfs = []
for i in range(1, 39):
    articles = browser.find_elements_by_tag_name('li[class=&quot;arxiv-result&quot;]')

    for article in articles:
        title = article.find_element_by_tag_name('p[class=&quot;title is-5 mathjax&quot;]').text
        arxiv_id = article.find_element_by_tag_name('a').text.replace('arXiv:','')
        arxiv_link = article.find_elements_by_tag_name('a')[0].get_attribute('href') 
        pdf_link = article.find_elements_by_tag_name('a')[1].get_attribute('href')
        authors = article.find_element_by_tag_name('p[class=&quot;authors&quot;]').text.replace('Authors:','')

        try:
                link1 = browser.find_element_by_link_text('▽ More')
                link1.click()
        except:
                time.sleep(0.1)

        abstract = article.find_element_by_tag_name('p[class=&quot;abstract mathjax&quot;]').text
        date = article.find_element_by_tag_name('p[class=&quot;is-size-7&quot;]').text
        date = re.split(r&quot;Submitted|;&quot;,date)[1]
        tag = article.find_element_by_tag_name('div[class=&quot;tags is-inline-block&quot;]').text.replace('\n', ',')
        
        try:
            doi = article.find_element_by_tag_name('div[class=&quot;tags has-addons&quot;]').text
            doi = re.split(r'\s', doi)[1] 
        except NoSuchElementException:
            doi = 'None'

        all_combined = [title, arxiv_id, arxiv_link, pdf_link, authors, abstract, date, tag, doi]
        dfs.append(all_combined)

    print('Finished Extracting Page:', i)

    try:
        link2 = browser.find_element_by_class_name('pagination-next')
        link2.click()
    except:
        browser.close
        
    time.sleep(0.1)


</code></pre>