title: XPath returns empty list. Why is it ignoring targeted div element?
body: <p>I'm a newbee to XPath and Scrapy. I'm trying to target a node which does not have a unique class (i.e. <code>class="pubBody"</code>). </p>
  
  <p>Already tried: 
  <a href="https://stackoverflow.com/questions/28163626/xpath-not-contains-a-and-b">xpath not contains A and B</a></p>
  
  <p>This should be a simple task but XPath just misses the second item. I am doing this from the scrapy shell. On the command prompt:</p>
  
  <p>scrapy shell "<a href="http://www.sciencedirect.com/science/journal/00221694/" rel="nofollow noreferrer">http://www.sciencedirect.com/science/journal/00221694/</a>"</p>
  
  <p>I am looking for the second div:</p>
  
  <pre><code>&lt;div id="issueListHeader" class="pubBody"&gt;...&lt; /div&gt;
  
  &lt;div class="pubBody"&gt;... &lt; /div&gt; 
  </code></pre>
  
  <p>I can only get the first but not the second. The best answers to similar questions suggested trying something like:</p>
  
  <pre><code>hxs.xpath('//div[contains(@class,"pubBody") and not(contains(@id,"issueListHeader"))]') 
  </code></pre>
  
  <p>but this returns an empty list for some reason. Any help please? Must be missing something silly, I've tried this for days!</p>
  
  <p>Other details:</p>
  
  <p>Once in the scrapy shell:</p>
  
  <pre><code>import scrapy
  
  xs = scrapy.Selector(response)
  
  hxs.xpath('//div[@class="pubBody"]')
  </code></pre>
  
  <p>Which works only for the first div element:</p>
  
  <pre><code>[&lt;Selector xpath='//div[@class="pubBody"]' data='&lt;div id="issueListHeader" class="pubBody'&gt;]
  </code></pre>
  
  <p>For the failed second div element I've also tried:</p>
  
  <pre><code>hxs.xpath('//div[@class="pubBody" and not(@id="issueListHeader")]').extract_first()
  
  hxs.xpath('//div[starts-with(@class, "pubBody") and not(re:test(@id, "issueListHeader"))]')
  </code></pre>
  
  <p>Also directly copied the XPath from Chrome, but also returns '[]':</p>
  
  <pre><code>hxs.xpath('//*[@id="issueList"]/div/form/div[2]')
  </code></pre>