<p>I think you should try in your scrapy shell,for experimentation.
1. scrapy shell '<a href="http://export.arxiv.org/rss/hep-th/recent" rel="nofollow noreferrer">http://export.arxiv.org/rss/hep-th/recent</a>'</p>

<ol start="2">
<li><p>sel.remove_namespaces()</p></li>
<li><p>a = sel.xpath('//title/text()')</p></li>
</ol>

<p><img src="https://i.stack.imgur.com/Zpmty.jpg" alt="enter image description here"></p>
scraping arXiv xml data with Scrapy
<p>I'm trying to get information from arXiv's page with <a href="http://scrapy.org/" rel="nofollow">scrapy</a> but cannot select "items" from their <a href="http://export.arxiv.org/rss/hep-th/recent" rel="nofollow">xml page</a>:</p>

<pre><code>from scrapy.spider import BaseSpider
from scrapy.selector import XmlXPathSelector

class arXivSpider(BaseSpider):
    name = "arxiv"
    allowed_domains = ["arxiv.org"]
    start_urls = ["http://export.arxiv.org/rss/hep-th/recent"]

    def parse(self, response):
        xxs = XmlXPathSelector(response)
        papers = xxs.select('//item')
        print papers
</code></pre>

<p>The item object is pretty straightforward, if I could extract it...</p>

<pre><code>&lt;item rdf:about="http://arxiv.org/abs/1112.5754"&gt;
&lt;title&gt;blah blah ... blah&lt;/title&gt;
&lt;link&gt;http://arxiv.org/abs/1112.5754&lt;/link&gt;
&lt;description rdf:parseType="Literal"&gt;&lt;p&gt;...&lt;/p&gt;&lt;/description&gt;
&lt;dc:creator&gt;blah, blah blah&lt;/dc:creator&gt;
&lt;/item&gt;
</code></pre>

<p>The script runs perfectly, it's just <code>papers = []</code> so the spider is not collecting <code>item</code>'s.  It may have to do w/ namespaces... </p>