title: Is there a way to access information from behind a dropdown menu with requests in python?
body: <p>I am trying to scrap some information off of a website that has a dropdown menu and once you click one of the options it brings you to a new page with the same link and I am unsure of how to get to this information with the requests library.
  here is the HTML of the dropdown.
  Would love some help
  Thank you!</p>
  <pre class="lang-html prettyprint-override"><code>&lt;ul class=&quot;list&quot;&gt;&lt;li data-value=&quot;&quot; class=&quot;option selected focus&quot;&gt;Select a Market&lt;/li&gt;&lt;li data-value=&quot;Show All&quot; class=&quot;option&quot;&gt;Show All&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfe6&quot; class=&quot;option&quot;&gt;US - S&amp;amp;P 500 Index&lt;/li&gt;&lt;li data-value=&quot;5a2b420decfdc711085dc51d&quot; class=&quot;option&quot;&gt;Bitcoin Per USD&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfeb&quot; class=&quot;option&quot;&gt;US - Dow Jones Industrials Index&lt;/li&gt;&lt;li data-value=&quot;5a8e2222ecfdc7197c27d1a3&quot; class=&quot;option&quot;&gt;US - CBOE VIX Index&lt;/li&gt;&lt;li data-value=&quot;5653923cc7eb6e355878c00d&quot; class=&quot;option&quot;&gt;US Dollar Index (Cash)&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf37&quot; class=&quot;option&quot;&gt;NY Gold Futures&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878beb4&quot; class=&quot;option&quot;&gt;NY Crude Oil Futures&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878beb1&quot; class=&quot;option&quot;&gt;NY Natural Gas Futures&lt;/li&gt;&lt;li data-value=&quot;5653922fc7eb6e355878bed6&quot; class=&quot;option&quot;&gt;Canadian Dollar Spot&lt;/li&gt;&lt;li data-value=&quot;5653922cc7eb6e355878be8c&quot; class=&quot;option&quot;&gt;Wheat CBT Futures&lt;/li&gt;&lt;li data-value=&quot;5ff35e7a54e89812b492df70&quot; class=&quot;option&quot;&gt;Uranium Spot Per Pound&lt;/li&gt;&lt;li data-value=&quot;56539230c7eb6e355878beda&quot; class=&quot;option&quot;&gt;Euro Adjusted Spot&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf36&quot; class=&quot;option&quot;&gt;NY Copper Futures&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf32&quot; class=&quot;option&quot;&gt;NY Platinum Futures&lt;/li&gt;&lt;li data-value=&quot;5653922cc7eb6e355878be93&quot; class=&quot;option&quot;&gt;Soybeans CBT Futures&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf34&quot; class=&quot;option&quot;&gt;NY Silver COMEX Futures&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bff3&quot; class=&quot;option&quot;&gt;US - PHLX Gold/Silver Index&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfc9&quot; class=&quot;option&quot;&gt;US - Oil &amp;amp; Gas Stock Index&lt;/li&gt;&lt;li data-value=&quot;5653922cc7eb6e355878be9e&quot; class=&quot;option&quot;&gt;Corn (CBOT)&lt;/li&gt;&lt;li data-value=&quot;5a8e221fecfdc7197c27d189&quot; class=&quot;option&quot;&gt;Global - MSCI World $ Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221eecfdc7197c27d186&quot; class=&quot;option&quot;&gt;Global - MSCI Emerging Markets $ Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221eecfdc7197c27d184&quot; class=&quot;option&quot;&gt;Global - MSCI All Country World $ Index&lt;/li&gt;&lt;li data-value=&quot;5a8e2221ecfdc7197c27d19a&quot; class=&quot;option&quot;&gt;Europe - EURO STOXX 50 Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221aecfdc7197c27d15d&quot; class=&quot;option&quot;&gt;US - NASDAQ 100 Index&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878c005&quot; class=&quot;option&quot;&gt;US - NASDAQ Composite Index&lt;/li&gt;&lt;li data-value=&quot;5c3ee86f2bbac10948c4c48c&quot; class=&quot;option&quot;&gt;US - Russell 1000 Index&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfcd&quot; class=&quot;option&quot;&gt;US - Russell 2000 Index&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfec&quot; class=&quot;option&quot;&gt;US - Dow Jones Utilities Index&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfda&quot; class=&quot;option&quot;&gt;Canada - S&amp;amp;P/TSX Composite Index&lt;/li&gt;&lt;li data-value=&quot;5a8e2220ecfdc7197c27d196&quot; class=&quot;option&quot;&gt;Canada - S&amp;amp;P / TSE 60 Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221fecfdc7197c27d18c&quot; class=&quot;option&quot;&gt;US - New York Stock Exchange Composite Index&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878c003&quot; class=&quot;option&quot;&gt;Mexico - IPC Index&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfe9&quot; class=&quot;option&quot;&gt;Germany - DAX Index&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfe3&quot; class=&quot;option&quot;&gt;France - CAC-40 Index&lt;/li&gt;&lt;li data-value=&quot;5653923ac7eb6e355878bfef&quot; class=&quot;option&quot;&gt;UK - FTSE 100 Index&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfc8&quot; class=&quot;option&quot;&gt;Spain - IBEX 35 Index&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878c007&quot; class=&quot;option&quot;&gt;Japan - NIKKEI Tokyo Index&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878c000&quot; class=&quot;option&quot;&gt;South Korea - KOSPI 200 Index&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfd0&quot; class=&quot;option&quot;&gt;China - Shanghai Composite&lt;/li&gt;&lt;li data-value=&quot;56539239c7eb6e355878bfd1&quot; class=&quot;option&quot;&gt;China - Shenzhen Composite&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878bff6&quot; class=&quot;option&quot;&gt;Hong Kong - Hang Seng Cash Share Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221fecfdc7197c27d18b&quot; class=&quot;option&quot;&gt;India - Nifty 50 Index&lt;/li&gt;&lt;li data-value=&quot;5653923bc7eb6e355878bffa&quot; class=&quot;option&quot;&gt;India - S&amp;amp;P BSE SENSEX Index&lt;/li&gt;&lt;li data-value=&quot;5a8e221aecfdc7197c27d15f&quot; class=&quot;option&quot;&gt;Russia - Micex Index&lt;/li&gt;&lt;li data-value=&quot;56539230c7eb6e355878bedb&quot; class=&quot;option&quot;&gt;Japanese Yen Spot&lt;/li&gt;&lt;li data-value=&quot;5653922fc7eb6e355878bed5&quot; class=&quot;option&quot;&gt;British Pound Spot&lt;/li&gt;&lt;li data-value=&quot;5653922fc7eb6e355878bed4&quot; class=&quot;option&quot;&gt;Australian Dollar Spot&lt;/li&gt;&lt;li data-value=&quot;5653922ec7eb6e355878bec3&quot; class=&quot;option&quot;&gt;New Zealand Dollar&lt;/li&gt;&lt;li data-value=&quot;56539230c7eb6e355878bedc&quot; class=&quot;option&quot;&gt;Swiss Franc Spot&lt;/li&gt;&lt;li data-value=&quot;5653922ec7eb6e355878becc&quot; class=&quot;option&quot;&gt;Brazil Real Spot&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878bebd&quot; class=&quot;option&quot;&gt;Chinese Yuan Spot&lt;/li&gt;&lt;li data-value=&quot;56539230c7eb6e355878bee5&quot; class=&quot;option&quot;&gt;Mexico Peso Spot&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878beba&quot; class=&quot;option&quot;&gt;Russian Rubles Spot&lt;/li&gt;&lt;li data-value=&quot;5653923fc7eb6e355878c073&quot; class=&quot;option&quot;&gt;South African Rand&lt;/li&gt;&lt;li data-value=&quot;5653922fc7eb6e355878bed2&quot; class=&quot;option&quot;&gt;Hong Kong Dollar Spot&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878bebc&quot; class=&quot;option&quot;&gt;Turkish Lira Spot&lt;/li&gt;&lt;li data-value=&quot;5653922ec7eb6e355878bec5&quot; class=&quot;option&quot;&gt;Singapore Dollar Spot&lt;/li&gt;&lt;li data-value=&quot;5653922ec7eb6e355878bec6&quot; class=&quot;option&quot;&gt;South Korean Won Spot&lt;/li&gt;&lt;li data-value=&quot;5b5b72237be3b20794c6dc51&quot; class=&quot;option&quot;&gt;US BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b721e7be3b20794c6dc0c&quot; class=&quot;option&quot;&gt;Canada BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b721e7be3b20794c6dc18&quot; class=&quot;option&quot;&gt;Euro BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b72237be3b20794c6dc4e&quot; class=&quot;option&quot;&gt;UK BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b721d7be3b20794c6dc03&quot; class=&quot;option&quot;&gt;Germany BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b72207be3b20794c6dc2c&quot; class=&quot;option&quot;&gt;Japan BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b72207be3b20794c6dc29&quot; class=&quot;option&quot;&gt;Italy BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b721e7be3b20794c6dc15&quot; class=&quot;option&quot;&gt;Spain BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b721c7be3b20794c6dbfd&quot; class=&quot;option&quot;&gt;Australia BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;5b5b72207be3b20794c6dc32&quot; class=&quot;option&quot;&gt;Mexico BMK 10 Yr Index&lt;/li&gt;&lt;li data-value=&quot;56539237c7eb6e355878bf9e&quot; class=&quot;option&quot;&gt;Gld Miners&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf30&quot; class=&quot;option&quot;&gt;NY Palladium Futures&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878bea9&quot; class=&quot;option&quot;&gt;US CBT 10 Year T-Notes&lt;/li&gt;&lt;li data-value=&quot;5653922dc7eb6e355878bec2&quot; class=&quot;option&quot;&gt;Norwegian Krone Spot&lt;/li&gt;&lt;li data-value=&quot;5c4171982bbac1517cff4b0b&quot; class=&quot;option&quot;&gt;US - iShares China Large-Cap ETF&lt;/li&gt;&lt;li data-value=&quot;5653923cc7eb6e355878c018&quot; class=&quot;option&quot;&gt;US - United States Oil Fund LP&lt;/li&gt;&lt;li data-value=&quot;5c41719c2bbac1517cff4b39&quot; class=&quot;option&quot;&gt;US - SPDR S&amp;amp;P Oil &amp;amp; Gas Exploration &amp;amp; Production ETF&lt;/li&gt;&lt;li data-value=&quot;5c41719b2bbac1517cff4b2e&quot; class=&quot;option&quot;&gt;US - Financial Select Sector SPDR Fund&lt;/li&gt;&lt;li data-value=&quot;5c41719b2bbac1517cff4b2d&quot; class=&quot;option&quot;&gt;US - Energy Select Sector SPDR Fund&lt;/li&gt;&lt;li data-value=&quot;5c41719c2bbac1517cff4b38&quot; class=&quot;option&quot;&gt;US - SPDR S&amp;amp;P Oil &amp;amp; Gas Equipment &amp;amp; Services ETF&lt;/li&gt;&lt;li data-value=&quot;56539232c7eb6e355878bf0b&quot; class=&quot;option&quot;&gt;US - Chevron Corporation&lt;/li&gt;&lt;li data-value=&quot;5653923dc7eb6e355878c03e&quot; class=&quot;option&quot;&gt;US - Alphabet, Inc - Google C shares &lt;/li&gt;&lt;li data-value=&quot;56539231c7eb6e355878bef7&quot; class=&quot;option&quot;&gt;US - Tesla Motors&lt;/li&gt;&lt;li data-value=&quot;5e9f388054e898058c527d96&quot; class=&quot;option&quot;&gt;CA - Vermilion Energy Inc&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf44&quot; class=&quot;option&quot;&gt;Ur Energy NC&lt;/li&gt;&lt;li data-value=&quot;56539237c7eb6e355878bf95&quot; class=&quot;option&quot;&gt;US - Freeport-McMoRan&lt;/li&gt;&lt;li data-value=&quot;5653923ec7eb6e355878c060&quot; class=&quot;option&quot;&gt;US - NetFlix Inc.&lt;/li&gt;&lt;li data-value=&quot;6011ffa954e898481c0ca143&quot; class=&quot;option&quot;&gt;US - Gamestop&lt;/li&gt;&lt;li data-value=&quot;5ec2bae254e8982e6ca74a47&quot; class=&quot;option&quot;&gt;US - AMC Inc. &lt;/li&gt;&lt;li data-value=&quot;5c4171ab2bbac1517cff4ba5&quot; class=&quot;option&quot;&gt;US - Under Armour - Class A&lt;/li&gt;&lt;li data-value=&quot;5c4171a62bbac1517cff4b85&quot; class=&quot;option&quot;&gt;US - Qualcomm&lt;/li&gt;&lt;li data-value=&quot;56539232c7eb6e355878bf17&quot; class=&quot;option&quot;&gt;US - General Electric&lt;/li&gt;&lt;li data-value=&quot;5653923ec7eb6e355878c050&quot; class=&quot;option&quot;&gt;UK - Royal Dutch Shell&lt;/li&gt;&lt;li data-value=&quot;56539233c7eb6e355878bf27&quot; class=&quot;option&quot;&gt;US - Marathon Oil&lt;/li&gt;&lt;li data-value=&quot;5a8e2223ecfdc7197c27d1b0&quot; class=&quot;option&quot;&gt;US - PHLX Oil Service Index&lt;/li&gt;&lt;li data-value=&quot;5653923cc7eb6e355878c01a&quot; class=&quot;option&quot;&gt;US - Invesco QQQ Trust&lt;/li&gt;&lt;li data-value=&quot;56539231c7eb6e355878befb&quot; class=&quot;option&quot;&gt;US - Amazon&lt;/li&gt;&lt;li data-value=&quot;56539234c7eb6e355878bf63&quot; class=&quot;option&quot;&gt;Suncor&lt;/li&gt;&lt;li data-value=&quot;56539230c7eb6e355878bee8&quot; class=&quot;option&quot;&gt;US - ExxonMobil&lt;/li&gt;&lt;li data-value=&quot;56539235c7eb6e355878bf73&quot; class=&quot;option&quot;&gt;Agnico Eagle Mines Limited&lt;/li&gt;&lt;li data-value=&quot;56539238c7eb6e355878bfc0&quot; class=&quot;option&quot;&gt;Mcewen Min&lt;/li&gt;&lt;li data-value=&quot;56539235c7eb6e355878bf68&quot; class=&quot;option&quot;&gt;Teck&lt;/li&gt;&lt;li data-value=&quot;56539236c7eb6e355878bf84&quot; class=&quot;option&quot;&gt;Cameco&lt;/li&gt;&lt;li data-value=&quot;5c4171ad2bbac1517cff4bb5&quot; class=&quot;option&quot;&gt;Canadian Residential Real Estate Index&lt;/li&gt;&lt;li data-value=&quot;5c4171ae2bbac1517cff4bc6&quot; class=&quot;option&quot;&gt;US Residential Real Estate Index&lt;/li&gt;&lt;/ul&gt;
  </code></pre>