title: Access next sibling <li> element with BeautifulSoup
body: <p>I am completely new to web parsing with Python/BeautifulSoup. I have an HTML that has (part of) the code as follows:</p>
  
  <pre><code>&lt;div id="pages"&gt;
      &lt;ul&gt;
          &lt;li class="active"&gt;&lt;a href="example.com"&gt;Example&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href="example.com"&gt;Example&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href="example1.com"&gt;Example 1&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href="example2.com"&gt;Example 2&lt;/a&gt;&lt;/li&gt;
      &lt;/ul&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>I have to visit <strong>each link</strong> (basically each <code>&lt;li&gt;</code> element) until there are no more <code>&lt;li&gt;</code> tags present. Each time a link is clicked, its corresponding <code>&lt;li&gt;</code> element gets class as 'active'. My code is:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import urllib2
  import re
  
  landingPage = urllib2.urlopen('somepage.com').read()
  soup = BeautifulSoup(landingPage)
  
  pageList = soup.find("div", {"id": "pages"})
  
  page = pageList.find("li", {"class": "active"})
  </code></pre>
  
  <p>This code gives me the first <code>&lt;li&gt;</code> item in the list. My logic is I am keeping on checking if the <code>next_sibling</code> is not None. If it is not None, I am creating an HTTP request to the <code>href</code> attribute of the <code>&lt;a&gt;</code> tag in that sibling <code>&lt;li&gt;</code>. That would get me to the next page, and so on, till there are no more pages. </p>
  
  <p>But I can't figure out how to get the <code>next_sibling</code> of the <code>page</code> variable given above. Is it <code>page.next_sibling.get("href")</code> or something like that? I looked through the documentation, but somehow couldn't find it. Can someone help please? </p>