title: Python - Issues trying to use LXML to parse Website Search Results
body: <p>I am trying to use LXML to parse the search results returned from this search URL:</p>
  
  <pre><code>http://www.rte.ie/player/ie/search/?q=news
  </code></pre>
  
  <p>The article tags returned in the HTML is this:</p>
  
  <pre><code>  &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117771/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/0005d4bf-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117771/"&gt;elev8&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117771/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;Ivan and Sean talk to future basketball sensation Julian Newman and the &lt;span class="search-highlight"&gt;News&lt;/span&gt; Dudes are in the loft with some crazy &lt;span class="search-highlight"&gt;news&lt;/span&gt; stories.&lt;/p&gt;
       &lt;span
      class="sprite logo-rte-two search-channel-icon"&gt;RTÃ‰ 2&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10118015/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/000716b2-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10118015/"&gt;One &lt;span class="search-highlight"&gt;News&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10118015/"&gt;Wed 06 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;06/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;The One O'Clock &lt;span class="search-highlight"&gt;News&lt;/span&gt; followed by Weather.&lt;/p&gt;
      &lt;span class="sprite logo-rte-one search-channel-icon"&gt;RTÃ‰ 1&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117836/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/00071614-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117836/"&gt;&lt;span class="search-highlight"&gt;News&lt;/span&gt; on Two and World Forecast&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117836/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;All the &lt;span class="search-highlight"&gt;news&lt;/span&gt; and sport from home and abroad.&lt;/p&gt;
       &lt;span
      class="sprite logo-rte-two search-channel-icon"&gt;RTÃ‰ 2&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117816/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/000715f2-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117816/"&gt;Nine &lt;span class="search-highlight"&gt;News&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117816/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;The Nine &lt;span class="search-highlight"&gt;News&lt;/span&gt; followed by Weather.&lt;/p&gt;
      &lt;span class="sprite logo-rte-one search-channel-icon"&gt;RTÃ‰ 1&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117789/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/000715ae-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117789/"&gt;Six One &lt;span class="search-highlight"&gt;News&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117789/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;The Six One &lt;span class="search-highlight"&gt;News&lt;/span&gt; and Sport followed by Weather.&lt;/p&gt;
      &lt;span class="sprite logo-rte-one search-channel-icon"&gt;RTÃ‰ 1&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117784/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/000715a0-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117784/"&gt;Nuacht and &lt;span class="search-highlight"&gt;News&lt;/span&gt; with Signing&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117784/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;Nuacht and &lt;span class="search-highlight"&gt;News&lt;/span&gt; with Signing.&lt;/p&gt;
      &lt;span class="sprite logo-rte-one search-channel-icon"&gt;RTÃ‰ 1&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117770/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/0007158d-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117770/"&gt;&lt;span class="search-highlight"&gt;News&lt;/span&gt;2Day&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117770/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;Domestic and international &lt;span class="search-highlight"&gt;news&lt;/span&gt; items of interest to younger viewers.&lt;/p&gt;
       &lt;span
      class="sprite logo-rte-two search-channel-icon"&gt;RTÃ‰ 2&lt;/span&gt;
    &lt;/article&gt;
  
  
  
    &lt;article class="search-result clearfix"&gt;&lt;a
      href="/player/ie/show/10117728/" class="thumbnail-programme-link"&gt;&lt;span
          class="sprite thumbnail-icon-play"&gt;Watch Now&lt;/span&gt;&lt;img class="thumbnail" alt="Watch Now"
          src="http://img.rasset.ie/0007154e-261.jpg"&gt;&lt;/a&gt;
      &lt;h3 class="search-programme-title"&gt;&lt;a href="/player/ie/show/10117728/"&gt;One &lt;span class="search-highlight"&gt;News&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
      &lt;p class="search-programme-episodes"&gt;&lt;a href="/player/ie/show/10117728/"&gt;Tue 05 Mar 2013&lt;/a&gt;&lt;/p&gt;
      &lt;!-- p class="search-programme-date"&gt;05/03/2013&lt;/p --&gt;
      &lt;p class="search-programme-description"&gt;The One O'Clock &lt;span class="search-highlight"&gt;News&lt;/span&gt; followed by Weather.&lt;/p&gt;
      &lt;span class="sprite logo-rte-one search-channel-icon"&gt;RTÃ‰ 1&lt;/span&gt;
    &lt;/article&gt;
  </code></pre>
  
  <p>I have added the following code to try and parse the returned results but my issues are that the retruned results are not consistent. The sections I'm interested in are the repeating article tags but the problem is anywhere is finds the search text in the returned results is add the tag span class="search-highlight" and this is throwing out my parsing.</p>
  
  <pre><code>url = "http://www.rte.ie/player/ie/search/?q=news"
  req = urllib2.Request(url)
  req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3 Gecko/2008092417 Firefox/3.0.3')
  response = urllib2.urlopen(req)
  html = str(response.read())
  response.close()
  
  parser = etree.HTMLParser(encoding='utf-8')
  tree   = etree.fromstring(html, parser)
  
  for elem in tree.xpath('//article[@class="search-result clearfix"]'):
      icon_url = str(elem[0][1].attrib.get('src'))
      print 'icon_url ', icon_url
  
      name_tmp = str(elem[1][0].text)
      print 'name_tmp ', name_tmp
  
      stream = str(elem[1][0].attrib.get('href'))
      print 'stream ', stream
  
      date_tmp = str(elem[2][0].text)
      print 'date_tmp ', date_tmp
  
      short_tmp = elem[4].text
      print 'short_tmp ', short_tmp
  
      channel =  elem[5].text
      print 'channel ', channel
  </code></pre>
  
  <p>The problem fields are name_tmp and short_tmp, they are dropping the full text names because of the search-highlight span tags. Can anyone think of a way to parse the full text or to ignore the span tags?</p>
  
  <p>Sorry for the very long post...</p>