title: Why the DupeFilter and RFPDupefilter are not middleware objects?
body: <p>While working on my crawler, I needed to implement a persistent duplicate filter that will detect already scraped items not only to drop them, but also to create a new relationship in DB. I actually store both items and researches (N to N relationship), thus if an items is already knew (I don't care about the update), I still want it to be associated to my new research.</p>
  
  <p>Anyway, I found a solution by overriding the RFPDupeFilter.</p>
  
  <p>Yet, I spent hours digging into Spider Middleware and Downloader Middleware as I was convinced the solution was to implement a new middleware - the scrapy architecture shared here seems straightforward <a href="https://doc.scrapy.org/en/latest/topics/architecture.html#data-flow" rel="nofollow noreferrer">https://doc.scrapy.org/en/latest/topics/architecture.html#data-flow</a>.
  And finally I saw references to DupeFilter and RFPDupeFilter that is indeed exactly what I need. </p>
  
  <p>Question is: why those filters are not implemented as middlewares? What makes those filter so specific that they couldn't fit the middleware interface?</p>
  
  <p>Thank you for the clarification</p>