title: How to scrape the links hidden in the dropdown menu using scrapy?
body: <p>I am going to scrape NFL depth charts from web pages, e.g. <a href="https://www.ourlads.com/nfldepthcharts/archive/220/BUF" rel="nofollow noreferrer">https://www.ourlads.com/nfldepthcharts/archive/220/BUF</a> . Now I want to get all the links to this kind of pages, but the source code of the dropdown menu "Archive Dates" does not include any links:</p>
  
  <pre class="lang-html prettyprint-override"><code>    &lt;option value=""&gt;-- Archive Dates --&lt;/option&gt;
      &lt;option value="220"&gt;05/01/2019&lt;/option&gt;
      &lt;option value="219"&gt;04/01/2019&lt;/option&gt;
      &lt;option value="218"&gt;03/01/2019&lt;/option&gt;
      &lt;option value="217"&gt;02/01/2019&lt;/option&gt;
      &lt;option value="216"&gt;01/01/2019&lt;/option&gt;
      &lt;option value="215"&gt;12/01/2018&lt;/option&gt;
      &lt;option value="214"&gt;11/01/2018&lt;/option&gt;
      &lt;option value="213"&gt;10/01/2018&lt;/option&gt;
      &lt;option value="212"&gt;09/01/2018&lt;/option&gt;
      &lt;option value="211"&gt;08/01/2018&lt;/option&gt;
  </code></pre>
  
  <p>I read a post <a href="https://stackoverflow.com/questions/53091598/web-scrape-get-drop-down-menu-data-python">Web scrape get drop-down menu data python</a> , which I think is helpful since he indicated that the web page uses JavaScript. </p>
  
  <p>But that answer uses selenium. I wonder if I can solve the problem using scrapy or beautifulsoup.</p>
  
  <p>The following is the structure of my scraper.</p>
  
  <pre class="lang-py prettyprint-override"><code>class depth_chart_archive_spider(scrapy.Spider):
      name = "depth_chart_archive"
      start_urls = ('https://www.ourlads.com/nfldepthcharts/',)
  
      def parse(self, response):
          dchome = BeautifulSoup(response.body, 'html.parser')
  
          # get the links somehow
  
          for link in links:
              yield scrapy.Request(link, callback = self.parse_team)
  
      def parse_team(self, response):
          # parse the page
  </code></pre>