<p>According to the <a href="https://arxiv.org/help/api/user-manual#query_details" rel="nofollow noreferrer">arXiv documentation</a>, there is no <code>published</code> or <code>date</code> field available.</p>
<p>What you can do is to <a href="https://arxiv.org/help/api/user-manual#sort" rel="nofollow noreferrer">sort the results</a> by date (by adding <code>&amp;sortBy=submittedDate&amp;sortOrder=descending</code> to your query parameters) and stop making requests when you reach 2018.</p>
<p>Basically your code should be modified like this:</p>
<pre class="lang-py prettyprint-override"><code>import urllib.request

import time
import feedparser

# Base api query url
base_url = 'http://export.arxiv.org/api/query?';

# Search parameters
search_query = urllib.parse.quote(&quot;ti:machine learning&quot;)
i = 0
results_per_iteration = 1000
wait_time = 3
papers = []
year = &quot;&quot;  
print('Searching arXiv for %s' % search_query)

while (year != &quot;2018&quot;): #stop requesting when papers date reach 2018
    print(&quot;Results %i - %i&quot; % (i,i+results_per_iteration))
    
    query = 'search_query=%s&amp;start=%i&amp;max_results=%i&amp;sortBy=submittedDate&amp;sortOrder=descending' % (search_query,
                                                         i,
                                                         results_per_iteration)

    # perform a GET request using the base_url and query
    response = urllib.request.urlopen(base_url+query).read()

    # parse the response using feedparser
    feed = feedparser.parse(response)
    # Run through each entry, and print out information
    for entry in feed.entries:
        #print('arxiv-id: %s' % entry.id.split('/abs/')[-1])
        #print('Title:  %s' % entry.title)
        #feedparser v4.1 only grabs the first author
        #print('First Author:  %s' % entry.author)
        paper = {}
        paper[&quot;date&quot;] = entry.published
        year = paper[&quot;date&quot;][0:4]
        paper[&quot;title&quot;] = entry.title
        paper[&quot;first_author&quot;] = entry.author
        paper[&quot;summary&quot;] = entry.summary
        papers.append(paper)
    # Sleep a bit before calling the API again
    print('Bulk: %i' % 1)
    i += results_per_iteration
    time.sleep(wait_time)
</code></pre>
<p>for the &quot;post-filtering&quot; approach, once enough results are collected, I'd do something like this:</p>
<pre class="lang-py prettyprint-override"><code>papers2019 = [item for item in papers if item[&quot;date&quot;][0:4] == &quot;2019&quot;]
</code></pre>
How to query arXiv for a specific year?
<p>I'm using the code shown below in order to retrieve papers from arXiv. I want to retrieve papers that have words &quot;machine&quot; and &quot;learning&quot; in the title. The number of papers is large, therefore I want to implement a slicing by year (<code>published</code>).</p>
<p>How can I request records of 2020 and 2019 in <code>search_query</code>? Please notice that I'm not interested in post-filtering.</p>
<pre><code>import urllib.request

import time
import feedparser

# Base api query url
base_url = 'http://export.arxiv.org/api/query?';

# Search parameters
search_query = urllib.parse.quote(&quot;ti:machine learning&quot;)
start = 0
total_results = 5000
results_per_iteration = 1000
wait_time = 3

papers = []

print('Searching arXiv for %s' % search_query)

for i in range(start,total_results,results_per_iteration):
    
    print(&quot;Results %i - %i&quot; % (i,i+results_per_iteration))
    
    query = 'search_query=%s&amp;start=%i&amp;max_results=%i' % (search_query,
                                                         i,
                                                         results_per_iteration)

    # perform a GET request using the base_url and query
    response = urllib.request.urlopen(base_url+query).read()

    # parse the response using feedparser
    feed = feedparser.parse(response)

    # Run through each entry, and print out information
    for entry in feed.entries:
        #print('arxiv-id: %s' % entry.id.split('/abs/')[-1])
        #print('Title:  %s' % entry.title)
        #feedparser v4.1 only grabs the first author
        #print('First Author:  %s' % entry.author)
        paper = {}
        paper[&quot;date&quot;] = entry.published
        paper[&quot;title&quot;] = entry.title
        paper[&quot;first_author&quot;] = entry.author
        paper[&quot;summary&quot;] = entry.summary
        papers.append(paper)
    
    # Sleep a bit before calling the API again
    print('Bulk: %i' % 1)
    time.sleep(wait_time)
</code></pre>