title: How to crawl all webpages on website up to certain depth?
body: <p>I have a website and I would like to find a webpage with information about job vacancies. There is only one page usually with such information. So I start crawling with website and I manage to get all webpages up to certain depth. It works. But they are many times duplicated. Instead of lets say 45 pages I get 1000 pages. I know the reason why. The reason is that every time I call my "parse" function, it parses all the webpages on a certain webpage. So when I come to a new webpage, it crawls all webpages, out of which some have been crawled before. </p>
  
  <p>1) I tried to make "items=[]" list out of parse function but I get some global error. I don't know how to get a list of unique webpages. When I have one, I will be able to choose the right one with simple url parsing. 
  2) I also tried to have "Request" and "return items" in the "parse" function, but I get syntax error: return inside generator.</p>
  
  <p>I am using DEPTH_LIMIT. Do I really need to use Rules ? </p>
  
  <p>code:</p>
  
  <pre><code>import scrapy, urlparse, os
  from scrapy.spiders import Rule
  from scrapy.linkextractors import LinkExtractor
  from tutorial.items import JobItem
  from scrapy.utils.response import get_base_url
  from scrapy.http import Request
  from urlparse import urljoin
  from datetime import datetime
  
  
  class JobSpider(scrapy.Spider):
      name = "jobs"
      allowed_domains = ["www.gen-i.si"]
      start_urls = ["http://www.gen-i.si"]
  
      def parse(self, response):
          response.selector.remove_namespaces() #
          urls = response.xpath('//@href').extract()#choose all "href", either new websites either webpages on our website
          items = []
  
  
          base_url = get_base_url(response) #base url
  
  
          for url in urls:
              #we need only webpages, so we remove all websites and urls with strange characters
              if (url[0:4] != "http") and not any(x in url for x in ['%', ':', '?', '&amp;']):
  
                  item = JobItem()
                  absolute_url = urlparse.urljoin(base_url,url)
                  item["link"] = absolute_url
                  if item not in items:
                      items.append(item)
  
                      yield item
                      yield Request(absolute_url, callback = self.parse)
          #return items
  </code></pre>