title: Specifying a html nest structure in Python
body: <p>I am trying to scrape data from the following website
  <a href="https://www.ecfr.gov/on/2022-04-08/title-21/chapter-I/subchapter-E/part-556/subpart-B/section-556.50" rel="nofollow noreferrer">https://www.ecfr.gov/on/2022-04-08/title-21/chapter-I/subchapter-E/part-556/subpart-B/section-556.50</a></p>
  <p>Note that there is a nest structure (Tolerances -&gt; Cattle -&gt; Liver and muscle). This is also one of many sections in this legislation.</p>
  <p>There is a &quot;Developer Tools&quot; option, but I am having trouble keeping the nested structure
  <a href="https://www.ecfr.gov/api/renderer/v1/content/enhanced/2022-04-08/title-21?part=556&amp;section=556.50" rel="nofollow noreferrer">https://www.ecfr.gov/api/renderer/v1/content/enhanced/2022-04-08/title-21?part=556&amp;section=556.50</a></p>
  <p>I would like to convert this html to a pandas dataframe, keeping the nest structure. For example:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>h4</th>
  <th>Indent-2</th>
  <th>Indent-3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Amprolium</td>
  <td>(1) Cattle</td>
  <td>(i) Liver, kidney, and muscle: 0.5 ppm.</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>The problem is that class &quot;Indent-3&quot; should be nested in &quot;Indent-2&quot;, which should be nested in &quot;h4&quot;. I can create the desired data by specifying each class name, but if I want to loop through the sections, I don't want to have to specify each class name.</p>
  <p>Is there a more general way (without specifying the class name) to produce the dataframe? This my code so far.</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  url = r&quot;https://www.ecfr.gov/api/renderer/v1/content/enhanced/2022-04-08/title-21?part=556&amp;section=556.50&quot;
  
  r = requests.get(url)
  soup = BeautifulSoup(r.content,&quot;xml&quot;)
  df =pd.DataFrame()
  
  title = soup.find(&quot;h4&quot;).text
  id2 = soup.find(&quot;div&quot;, attrs = {&quot;id&quot;:&quot;p-556.50(b)(1)&quot;}).find(attrs = {&quot;class&quot;:&quot;indent-2&quot;}).text
  id3 = soup.find(&quot;div&quot;, attrs = {&quot;id&quot;:&quot;p-556.50(b)(1)(i)&quot;}).find(attrs = {&quot;class&quot;:&quot;indent-3&quot;}).text
  
  df = pd.DataFrame(data = {&quot;h4&quot;:[title],
                            &quot;indent-2&quot;:[id2],
                            &quot;indent-3&quot;:[id3]})
  </code></pre>