title: Avoid Duplicate URL Crawling
body: <p>I coded a simple crawler.
  In the settings.py file, by referring to scrapy documentation, I used </p>
  
  <pre><code>DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'
  </code></pre>
  
  <p>If I stop the crawler and restart the crawler again, it is scraping the duplicate urls again.
  Am I doing something wrong?</p>