title: Use scrapy to get list of urls, and then scrape content inside those urls
body: <p>I need a Scrapy spider to scrape the following page (<a href="https://www.phidgets.com/?tier=1&amp;catid=64&amp;pcid=57" rel="noreferrer">https://www.phidgets.com/?tier=1&amp;catid=64&amp;pcid=57</a>) for each URL (30 products, so 30 urls) and then go into each product via that url and scrape the data inside. </p>
  
  <p>I have the second part working exactly as I want:</p>
  
  <pre><code>import scrapy
  
  class ProductsSpider(scrapy.Spider):
      name = "products"
      start_urls = [
          'https://www.phidgets.com/?tier=1&amp;catid=64&amp;pcid=57',
      ]
  
      def parse(self, response):
          for info in response.css('div.ph-product-container'):
              yield {
                  'product_name': info.css('h2.ph-product-name::text').extract_first(),
                  'product_image': info.css('div.ph-product-img-ctn a').xpath('@href').extract(),
                  'sku': info.css('span.ph-pid').xpath('@prod-sku').extract_first(),
                  'short_description': info.css('div.ph-product-summary::text').extract_first(),
                  'price': info.css('h2.ph-product-price &gt; span.price::text').extract_first(),
                  'long_description': info.css('div#product_tab_1').extract_first(),
                  'specs': info.css('div#product_tab_2').extract_first(),
              }
  
          # next_page = response.css('div.ph-summary-entry-ctn a::attr("href")').extract_first()
          # if next_page is not None:
          #     yield response.follow(next_page, self.parse)
  </code></pre>
  
  <p>But I don't know how to do the first part. As you will see I have the main page (<a href="https://www.phidgets.com/?tier=1&amp;catid=64&amp;pcid=57" rel="noreferrer">https://www.phidgets.com/?tier=1&amp;catid=64&amp;pcid=57</a>) set as the start_url. But how do I get it to populate the start_urls list with all 30 urls I need crawled? </p>