title: XPath: Select Current and Next Node's text by Current Node Attributes
body: <p>If this is a repeat question, I apologize, but I can't find another question either on SO or elsewhere that seems to handle what I need. Here is my question:</p>
  
  <p>I'm using <code>scrapy</code> to get some information out of <a href="http://www.utm.utoronto.ca/regcal/WEBLISTCOURSES1.html" rel="nofollow">this</a> webpage. For clarity, following is a block of the source code from that webpage, which is of interest to me:</p>
  
  <pre><code>&lt;p class="titlestyle"&gt;ANT101H5 Introduction to Biological Anthropology and Archaeology 
                          &lt;span class='distribution'&gt;(SCI)&lt;/span&gt;&lt;/p&gt; 
  
  &lt;span class='normaltext'&gt; 
  Anthropology is the global and holistic study of human biology and behaviour, and includes four subfields: biological anthropology, archaeology, sociocultural anthropology and linguistics. The material covered is  directed  to answering the question: What makes us human? This course is a survey of  biological  anthropology and  archaeology.  [&lt;span class='Helpcourse'
              onMouseover="showtip(this,event,'24 Lectures')"
              onMouseout="hidetip()"&gt;24L&lt;/span&gt;, &lt;span class='Helpcourse'
              onMouseover="showtip(this,event,'12 Tutorials')"
              onMouseout="hidetip()"&gt;12T&lt;/span&gt;]&lt;br&gt; 
  
  &lt;span class='title2'&gt;Exclusion: &lt;/span&gt;&lt;a href='javascript:OpenCourse("WEBCOURSENOTFOUND.html")'&gt;ANT100Y5&lt;/a&gt;&lt;br&gt;
  
  &lt;span class='title2'&gt;Prerequisite: &lt;/span&gt;&lt;a href='javascript:OpenCourse("WEBCOURSEANT102H5.pl?fv=1")'&gt;ANT102H5&lt;/a&gt;&lt;br&gt; 
  &lt;/span&gt;&lt;br/&gt;&lt;br/&lt;br/&gt; 
  </code></pre>
  
  <p>Almost all of the code on that page looks like the above block.</p>
  
  <p>From all of this, I need to grab:</p>
  
  <ol>
  <li>ANT101H5 Introduction to Biological Anthropology and Archaeology </li>
  <li>Exclusion: ANT100Y5</li>
  <li>Prerequisite: ANT102H5</li>
  </ol>
  
  <p>The problem is that <code>Exclusion:</code> is inside a <code>&lt;span class="title2"&gt;</code> and <code>ANT100Y5</code> is inside the following <code>&lt;a&gt;</code>.</p>
  
  <p>I don't seem to be able to grab both of them out of this source code. Currently, I have code that attempts (and fails) to grab <code>ANT100Y5</code> which looks like:</p>
  
  <pre><code>hxs = HtmlXPathSelector(response)
      sites = hxs.select("//*[(name() = 'p' and @class = 'titlestyle') or (name() = 'a' and @href and preceding-sibling::'//span/@class=title2')]")
  </code></pre>
  
  <p>I'd appreciate any help with this, even if it's a "you're blind for not seeing this other SO question which answers this perfectly" (in which case,  myself will vote to close this). I really am that much at my wits end.</p>
  
  <p>Thanks in advance</p>
  
  <p><strong>EDIT: Complete original code after changes suggested by @Dimitre</strong></p>
  
  <p>I'm using the following code:</p>
  
  <pre><code>class regcalSpider(BaseSpider):
      name = "disc"
      allowed_domains = ['www.utm.utoronto.ca']
      start_urls = ['http://www.utm.utoronto.ca/regcal/WEBLISTCOURSES1.html']
  
      def parse(self, response):
              items = []
              hxs = HtmlXPathSelector(response)
              sites = hxs.select("/*/p/text()[1] | \
                                (//span[@class='title2'])[1]/text() | \
                                (//span[@class='title2'])[1]/following-sibling::a[1]/text() | \
                                (//span[@class='title2'])[2]/text() | \
                                (//span[@class='title2'])[2]/following-sibling::a[1]/text()")
  
              for site in sites:
                      item = RegcalItem()
                      item['title'] = site.select("a/text()").extract()
                      item['link'] = site.select("a/@href").extract()
                      item['desc'] = site.select("text()").extract()
                      items.append(item)
              return items
  
              filename = response.url.split("/")[-2]
              open(filename, 'wb').write(response.body)
  </code></pre>
  
  <p>Which gives me this result:</p>
  
  <pre><code>[{"title": [], "link": [], "desc": []},
   {"title": [], "link": [], "desc": []},
   {"title": [], "link": [], "desc": []}]
  </code></pre>
  
  <p>This is not the output that I need. What am I doing wrong? Keep in mind that I'm running this script on <a href="http://www.utm.utoronto.ca/regcal/WEBLISTCOURSES1.html" rel="nofollow">this</a>, as mentioned.</p>