title: (Scrapy) How do you scrape all the external links on each website from a list of hundreds of websites (and run the whole thing on Zyte)?
body: <p>I am looking for some help regarding my Scrapy project.
  I want to use Scrapy to code a generic Spider that would crawl multiple websites from a list. I was hoping to have the list in a separate file, because it's quite large. For each website, the spider will navigate through internal links, and on each page, it will collect every external links.</p>
  <p>I believe there are too many websites to create one spider per website.
  I want to <strong>scrape</strong> only external links, meaning &quot;absolute&quot; links whose domain name is different from the domain of the website where the link is found (subdomain would still be internal links from my POV).</p>
  <p>Eventually, I want to export the results in a CSV with the following fields:</p>
  <ul>
  <li>domain of the website being crawled, (from the list)</li>
  <li>page_url (where the external link was found)</li>
  <li>external_link
  If the same external link is found several times on the same page, it is deduped. Not yet sure though, but I might want to dedup external links on the website scope too, at some point.</li>
  </ul>
  <p>At some point, I would also like to :</p>
  <ul>
  <li>also filter certain external links to not be considered such as facebook.com/... etc,</li>
  <li>to run the script from Zyte.com. I believe it constrains me to respect a certain structure for the code, rather than just a standalone script. Any suggestion on that aspect would really help too.</li>
  </ul>
  <p>After a lot of research, I found this reference: <a href="https://coderedirect.com/questions/369975/dynamic-rules-based-on-start-urls-for-scrapy-crawlspider" rel="nofollow noreferrer">https://coderedirect.com/questions/369975/dynamic-rules-based-on-start-urls-for-scrapy-crawlspider</a></p>
  <p>But it wasn't clear to me how I can make it because it's missing a full version of the code.</p>
  <p>So far, the code I developed is as below, but I am stuck, as it does not fulfill my needs:</p>
  <pre><code>import scrapy
  from scrapy.crawler import CrawlerProcess
  from scrapy import Request
  # import the link extractor
  from scrapy.linkextractors import LinkExtractor
  import os
  
  
  class LinksSpider(scrapy.Spider):
      name = 'publishers_websites'
      start_urls = ['https://loremipsum.io/']
      allowed_domains = ['loremipsum.io']
      custom_settings = {'USER_AGENT': &quot;Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36&quot;}
  
      try:
          os.remove('publishers_websites.txt')
      except OSError:
          pass
  
      custom_settings = {
          'CONCURRENT_REQUESTS': 2,
          'AUTO_THROTTLE_ENABLED': True
      }
  
      def __init__(self):
          self.link_extractor = LinkExtractor(unique=True)
  
      def parse(self, response):
          domain = 'https://loremipsum.io/'
          all_links = self.link_extractor.extract_links(response)
          for link in all_links:
              if domain not in link.url:
                  with open('publishers_websites.txt', 'a+') as f:
                      f.write(f&quot;\n{str(response.request.url), str(link.url)}&quot;)
  
              yield response.follow(url=link, callback=self.parse)
  
  if __name__ == &quot;__main__&quot;:
      process = CrawlerProcess()
      process.crawl(LinksSpider)
      process.start()
  
  </code></pre>
  <p>There aren't much answers for my problem, and my Python skills are not good enough to solve the problem by myself.</p>
  <p>I would be very grateful for any help I receive.</p>