title: how to get entire document in scrapy using hxs.select
body: <p>I've been at this for 12hrs and I'm hoping someone can give me a leg up.</p>
  
  <p>Here is my code all I want is to get the anchor and url of every link on a page as it crawls along. </p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from scrapy.utils.url import urljoin_rfc
  from scrapy.utils.response import get_base_url
  from urlparse import urljoin
  
  #from scrapy.item import Item
  from tutorial.items import DmozItem
  
  class HopitaloneSpider(CrawlSpider):
  name = 'dmoz'
  allowed_domains = ['domain.co.uk']
  start_urls = [
      'http://www.domain.co.uk'
  ]
  
  rules = (
      #Rule(SgmlLinkExtractor(allow='&gt;example\.org', )),
      Rule(SgmlLinkExtractor(allow=('\w+$', )), callback='parse_item', follow=True),
  )
  
  user_agent = 'Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))'
  
  def parse_item(self, response):
      #self.log('Hi, this is an item page! %s' % response.url)
  
      hxs = HtmlXPathSelector(response)
      #print response.url
      sites = hxs.select('//html')
      #item = DmozItem()
      items = []
  
      for site in sites: 
  
                     item = DmozItem()
                     item['title'] = site.select('a/text()').extract()
                     item['link'] = site.select('a/@href').extract()
  
                     items.append(item)
  
      return items
  </code></pre>
  
  <p>What I'm doing wrong... my eyes hurt now.</p>