title: Scrapy with deep navigation
body: <p>I'm new to web scraping, and I have a web scraping project.</p>
  <p>If I have a website to scrape and the architecture of the pages on the website as follows:</p>
  <pre><code>A &gt;a&gt;b&gt;c&gt;d
  B &gt;a&gt;b&gt;c&gt;d
  C &gt;a&gt;b&gt;c&gt;d
  D &gt;a&gt;b&gt;c&gt;d 
  </code></pre>
  <p>and I want to extract the text from all the &quot;d&quot; in all sections (A, B, C, D)</p>
  <p>this is my code, and it is for a single page, I have no idea how to make it for all the pages I have.</p>
  <pre><code>import scrapy
  
  class LowsSpider(scrapy.Spider):
      name = 'laws'
      start_urls = ['https://myurl.com'] #I hide the website url for privacy issues. 
  
      def parse(self, response):
          for divLawText in response.css('#divLawText&gt;div&gt;*::text').getall(): # the css selector here is to extract all the text from all elements under the main div &quot;divLawText&quot;
              yield {'lawText':divLawText}
  </code></pre>
  <p>is it possible with Scrapy to scrape deep navigation ? or there are other scrapers better?</p>