<p>Okay, so first of all there is no data :( so I am just taking a sample ID from <a href="https://pypi.org/project/semanticscholar/" rel="nofollow noreferrer">semanticscholar</a> documents. Looking at your code, there I can see plenty of mistakes:</p>
<ol>
<li>Don't always stick to <code>pd.DataFrame</code> for your work! Dataframe are great, but are also slow! You just need to get the ID from the <code>'paperIds-1975-2005-2015-2.tsv'</code> so you can either read the file using <code>file.readline()</code> or you can save the data into a list:</li>
</ol>
<pre class="lang-py prettyprint-override"><code>data = pd.read_csv('paperIds-1975-2005-2015-2.tsv', sep='\t', names=[&quot;id&quot;]).id.values
</code></pre>
<ol start="2">
<li>From the code flow, what I understand is you want to save the scraped data into a <em>single</em> CSV file, right? So, why are you appending the data and saving the file again and again? This makes the code like 100000s time slower!</li>
<li>I really don't understand the purpose of <code>time.sleep(60)</code> you have added. If there is some error, you should print and move on - why wait?</li>
<li>For checking the progress, you can use the <code>tqdm</code> <a href="https://www.geeksforgeeks.org/python-how-to-make-a-terminal-progress-bar-using-tqdm/" rel="nofollow noreferrer">library</a> which shows a nice progress bar for your code!</li>
</ol>
<p>Taking these into consideration, I have modified your code as follows:</p>
<pre class="lang-py prettyprint-override"><code>import pandas as pd
import semanticscholar as sch
from tqdm import tqdm as TQ # for progree-bar

data = ['10.1093/mind/lix.236.433', '10.1093/mind/lix.236.433'] # using list or np.ndarray looks more logical!
print(data)
&gt;&gt; ['10.1093/mind/lix.236.433', '10.1093/mind/lix.236.433']
</code></pre>
<p>Once you have done this, you can now go and scrape the data. Okay, before that <code>pandas DataFrame</code> is basically a dictionary with advanced features. So, for our purpose, we will first add all the information to the dictionary and then create the dataframe. I personally prefer this process - gives me more control, if there are any changes need to be done.</p>
<pre class="lang-py prettyprint-override"><code>cols = ['id', 'abstract', 'arxivId', 'authors', 'citationVelocity', 'citations',
    'corpusId', 'doi', 'fieldsOfStudy', 'influentialCitationCount', 'is_open_access',
    'is_publisher_licensed', 'paperId', 'references', 'title', 'topics', 'url', 'venue', 'year']

outputData = dict((k, []) for k in cols)

print(outputData)
{'id': [],
 'abstract': [],
 'arxivId': [],
 'authors': [],
 'citationVelocity': [],
 'citations': [],
 'corpusId': [],
 'doi': [],
 'fieldsOfStudy': [],
 'influentialCitationCount': [],
 'is_open_access': [],
 'is_publisher_licensed': [],
 'paperId': [],
 'references': [],
 'title': [],
 'topics': [],
 'url': [],
 'venue': [],
 'year': []}
</code></pre>
<p>Now you can simply fetch the data and save it into your dataframe as below:</p>
<pre class="lang-py prettyprint-override"><code>for _paperID in TQ(data):
    paper = sch.paper(_paperID, timeout = 10) # scrape the paper
    for key in cols:
        try:
            outputData[key].append(paper.get(key))
        except KeyError:
            outputData[key].append(None) # if there is no data, append none
            print(f&quot;{key} not Found for {_paperID}&quot;)

pd.DataFrame(outputData).to_csv('output_file_name.csv', index = False)
</code></pre>
<p>This is the output that I have obtained:
<a href="https://i.stack.imgur.com/eo3iv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eo3iv.png" alt="enter image description here" /></a></p>
Scraping data parallel + batch processing
<p>I am doing a task that requires scraping. I have a dataset with ids and for each id i need to scrape some new information. This dataset has around 4 million rows. Here is my code:</p>
<pre><code>import pandas as pd
import numpy as np
import semanticscholar as sch
import time

# dataset with ids
df = pd.read_csv('paperIds-1975-2005-2015-2.tsv', sep='\t', names=[&quot;id&quot;])

# columns that will be produced
cols = ['id', 'abstract', 'arxivId', 'authors', 
        'citationVelocity', 'citations', 
        'corpusId', 'doi', 'fieldsOfStudy', 
        'influentialCitationCount', 'is_open_access', 
        'is_publisher_licensed', 'paperId', 
        'references', 'title', 'topics', 
        'url', 'venue', 'year']

# a new dataframe that we will append the scraped results
new_df = pd.DataFrame(columns=cols)

# a counter so we know when every 100000 papers are scraped
c = 0      
i = 0
while i &lt; df.shape[0]:
    try:
        paper = sch.paper(df.id[i], timeout=10) # scrape the paper
        new_df = new_df.append([df.id[i]]+paper, ignore_index=True) # append to the new dataframe
        new_df.to_csv('abstracts_impact.csv', index=False) # save it 
        if i % 100000 == 0: # to check how much we did
            print(c)
            c += 1
        i += 1
    except:
        time.sleep(60)
</code></pre>
<p>The problem is that the dataset is pretty big and this approach is not working. I left it working for 2 days and it scraped around 100000 ids, and then suddenly just froze and all the data that was saved was just empty rows.
I was thinking that the best solution would be to parallelize and batch processing. I never have done this before and I am not familiar with these concepts. Any help would be appreciated. Thank you!</p>