How to select some urls with BeautifulSoup? <p>I want to scrape the following information except the last row and "class="Region" row:</p> <pre><code>... &lt;td&gt;7&lt;/td&gt; &lt;td bgcolor="" align="left" style=" width:496px"&gt;&lt;a class="xnternal" href="http://www.whitecase.com"&gt;White and Case&lt;/a&gt;&lt;/td&gt; &lt;td bgcolor="" align="left"&gt;New York&lt;/td&gt; &lt;td bgcolor="" align="left" class="Region"&gt;N/A&lt;/td&gt; &lt;td bgcolor="" align="left"&gt;1,863&lt;/td&gt; &lt;td bgcolor="" align="left"&gt;565&lt;/td&gt; &lt;td bgcolor="" align="left"&gt;1,133&lt;/td&gt; &lt;td bgcolor="" align="left"&gt;$160,000&lt;/td&gt; &lt;td bgcolor="" align="center"&gt;&lt;a class="xnternal" href="/nlj250/firmDetail/7"&gt; View Profile &lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class="small" bgcolor="#FFFFFF"&gt; ... </code></pre> <p>I tested with this handler:</p> <pre><code>class TestUrlOpen(webapp.RequestHandler): def get(self): soup = BeautifulSoup(urllib.urlopen("http://www.ilrg.com/nlj250/")) link_list = [] for a in soup.findAll('a',href=True): link_list.append(a["href"]) self.response.out.write("""&lt;p&gt;link_list: %s&lt;/p&gt;""" % link_list) </code></pre> <p>This works but it also get the "View Profile" link which I don't want:</p> <pre><code>link_list: [u'http://www.ilrg.com/', u'http://www.ilrg.com/', u'http://www.ilrg.com/nations/', u'http://www.ilrg.com/gov.html', ......] </code></pre> <p>I can easily remove the "u'http://www.ilrg.com/'" after scraping the site but it would be nice to have a list without it. What is the best way to do this? Thanks.</p>