<blockquote>
  <p>It may have to do w/ namespaces...</p>
</blockquote>

<p>Yes it is.</p>

<p>XmlXPathSelector have ability to handle namespaces, by registering them (<a href="http://doc.scrapy.org/en/latest/topics/selectors.html#xmlxpathselector-examples" rel="nofollow">examples in documentation</a>). In your case:</p>

<pre><code>$ scrapy shell http://export.arxiv.org/rss/hep-th/recent
In [1]: xxs.register_namespace('g', 'http://purl.org/rss/1.0/')

In [2]: xxs.namespaces
Out[2]: {'g': 'http://purl.org/rss/1.0/'}

In [3]: xxs.select('//item')
Out[3]: []

In [4]: xxs.select('//g:item')
Out[4]:
[&lt;XmlXPathSelector xpath='//g:item' data=u'&lt;item xmlns="http://purl.org/rss/1.0/" x'&gt;,
 &lt;XmlXPathSelector xpath='//g:item' data=u'&lt;item xmlns="http://purl.org/rss/1.0/" x'&gt;,
...
</code></pre>
scraping arXiv xml data with Scrapy
<p>I'm trying to get information from arXiv's page with <a href="http://scrapy.org/" rel="nofollow">scrapy</a> but cannot select "items" from their <a href="http://export.arxiv.org/rss/hep-th/recent" rel="nofollow">xml page</a>:</p>

<pre><code>from scrapy.spider import BaseSpider
from scrapy.selector import XmlXPathSelector

class arXivSpider(BaseSpider):
    name = "arxiv"
    allowed_domains = ["arxiv.org"]
    start_urls = ["http://export.arxiv.org/rss/hep-th/recent"]

    def parse(self, response):
        xxs = XmlXPathSelector(response)
        papers = xxs.select('//item')
        print papers
</code></pre>

<p>The item object is pretty straightforward, if I could extract it...</p>

<pre><code>&lt;item rdf:about="http://arxiv.org/abs/1112.5754"&gt;
&lt;title&gt;blah blah ... blah&lt;/title&gt;
&lt;link&gt;http://arxiv.org/abs/1112.5754&lt;/link&gt;
&lt;description rdf:parseType="Literal"&gt;&lt;p&gt;...&lt;/p&gt;&lt;/description&gt;
&lt;dc:creator&gt;blah, blah blah&lt;/dc:creator&gt;
&lt;/item&gt;
</code></pre>

<p>The script runs perfectly, it's just <code>papers = []</code> so the spider is not collecting <code>item</code>'s.  It may have to do w/ namespaces... </p>