title: Scrapy crawl only the base domain links (or first level links)
body: <p>I am implementing a crawl spider that crawls the webpage and look for the same domain links and then crawls all that same domain pages. so far i successfully implement the crawl spider that extract all the links from the web-page, here is the rule that gets all</p>
  
  <pre><code>Rule(SgmlLinkExtractor(allow=(r'', )),
           callback='parse_start_url', follow=False),
  </code></pre>
  
  <p>and then i apply a check in the callback function, that is something like this </p>
  
  <pre><code>def parse_start_url(self, response):
      sel = Selector(response)
  
      request_url = response.request.headers.get('referer', " ")
      if (request_url in response.url) or (request_url == " "):
  
          # retrieving the text from he body of the page
          mystring = ''.join(sel.xpath("//body//text()").extract()).strip()
  </code></pre>
  
  <p>the above if condition is working fine for me but the problem is that it goes to all the link available on a given page and decision is made in the <strong>callback function</strong> , i want the Rule (link-extractor) help so that it only crawl the page that has the same base domain.
  or we can say that the link of that given site only.</p>
  
  <p>any help will be highly appreciated</p>
  
  <p>thanks</p>