<p>Using BeautifulSoup:</p>

<pre><code>import requests
import json
from bs4 import BeautifulSoup as bs

url = 'http://www.arxiv-sanity.com/top?timefilter=year&amp;vfilter=all'
res = requests.get(url)
text = res.text
soup=bs(text, "html.parser")
extract=soup.select('script')[6]

target = extract.decode().split('var papers = ')[1]
target2 = target.replace("}, {","}xxx{").replace('[{','{').replace('}];','}')
final = target2.split('xxx')

for i in range(len(final)):
    if i == len(final)-1:
        last = final[i].split('var pid')[0]
        d = json.loads(last)        
        print(d['title'],d['link'],d['abstract'])
    else:
        d = json.loads(final[i])
        print(d['title'],d['link'],d['abstract'])
</code></pre>

<p>Sample output:</p>

<pre><code>BERT: Pre-training of Deep Bidirectional Transformers for Language       Understanding 
http://arxiv.org/abs/1810.04805v2 
We introduce a new language representation model called BERT, which stands
for Bidirectional Encoder Representations from Transformers. Unlike recent
language representation models, BERT is designed to pre-train deep
bidirectional representations from unlabeled text by jointly conditioning on
both left and right context in all layers...
</code></pre>

<p>etc.</p>
how to crawling arxiv sanity?
<p>I want crawling "link", "title" and "abstract"</p>

<p>how can I get crawling this?</p>

<p>I tried</p>

<pre><code>import requests
import json

url = 'http://www.arxiv-sanity.com/top?timefilter=year&amp;vfilter=all'
res = requests.get(url)
text = res.text
# print(text)

d = json.loads(text)
print(d['title'], d['link'], d['abstract'])
</code></pre>

<p>but <code>SONDecodeError: Expecting value: line 1 column 1 (char 0)</code> occur</p>