def process_item(self, item, spider):
        if item['loc'] in self.processed:
            raise scrapy.exceptions.DropItem("Duplicate URL found: %s."
                                             % item['loc'])
        else:
            self.processed.add(item['loc'])
            return item
Check if a URL was already found.
dataset/ETHPy150Open openstack/openstack-doc-tools/sitemap/generator/pipelines.py/IgnoreDuplicateUrls.process_item