<p>A regular expression along with <strong>findall()</strong> method can be used for finding all the intersting links form the given html content.</p>
<p>BeautifulSoup offers an easy way to  read table from html.</p>
<p>The above goal of reading pdf links form a table inside a given html content can be achieved by using regex along with BeautifulSoup.</p>
<p><strong>Working example using regex and along with BeatifulSoup</strong></p>
<pre><code># File name: find-pdf-links.py

import re
from bs4 import BeautifulSoup

htmlContent = &quot;&quot;&quot;
    &lt;h3&gt;Three-way classification&lt;/h3&gt;
    ...
    &lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;
    ...
    &lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;
    ....
    &lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;
    ......
    &lt;td&gt;&lt;a href=&quot;https://www.nyu.edu/projects/bowman/spinn.pdf&quot;&gt;
    ...
    &lt;td&gt;&lt;a href=&quot;http://arxiv.org/pdf/1511.06361v3.pdf&quot;&gt;
&quot;&quot;&quot;

# Read webpage
webPage = BeautifulSoup(htmlContent)

# Read table form the webpage
tableOfLinks = webPage.find(&quot;table&quot;)

print(&quot;PDF links:&quot;)
for link in tableOfLinks.findAll('a', attrs={'href': re.compile(&quot;^http://.*pdf$&quot;)}):
    print(link.get('href'))
</code></pre>
<p><strong>Output:</strong></p>
<pre><code>&gt; python find-pdf-links.py

PDF links:
http://nlp.stanford.edu/pubs/snli_paper.pdf
http://nlp.stanford.edu/pubs/snli_paper.pdf
http://nlp.stanford.edu/pubs/snli_paper.pdf
http://arxiv.org/pdf/1511.06361v3.pdf
</code></pre>
<p><strong>More information:</strong></p>
<p><a href="https://www.w3schools.com/python/python_regex.asp" rel="nofollow noreferrer">https://www.w3schools.com/python/python_regex.asp</a></p>
<p><a href="https://www.geeksforgeeks.org/python-check-url-string/" rel="nofollow noreferrer">https://www.geeksforgeeks.org/python-check-url-string/</a></p>
Extracting paper URLs from leaderboard HTML based on regular expression
<h1>Problem</h1>
<p>I am now moving into a natural language processing projects. Before I get my hands dirty, I plan to read other people's works on dataset, where they are organized as a <a href="https://nlp.stanford.edu/projects/snli/" rel="nofollow noreferrer">leaderboard</a> (see &quot;Three-way Classification&quot; section).</p>
<p>However, in order to download these papers, I need to manually click on each URL (there are about 50 of them), which is time-consuming. Therefore, I am trying to extract these URLs from HTML, which looks like following:</p>
<pre><code>&lt;h3&gt;Three-way classification&lt;/h3&gt;

&lt;blockquote&gt;
&lt;table class=&quot;newstuff&quot;&gt;

&lt;tr class=&quot;header&quot;&gt;
&lt;th&gt;Publication&lt;/th&gt;
&lt;th&gt;&amp;nbsp;Model&lt;/th&gt;
&lt;th&gt;Parameters&lt;/th&gt;
&lt;th&gt;&amp;nbsp;Train (% acc)&lt;/th&gt;
&lt;th&gt;&amp;nbsp;Test (% acc)&lt;/th&gt;
&lt;/tr&gt;

&lt;tr class=&quot;section&quot;&gt;
&lt;th colspan=&quot;5&quot; style=&quot;background-color:transparent; color:#646464;&quot;&gt;Feature-based models&lt;/th&gt;
&lt;/tr&gt;

&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;Bowman et al. '15&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Unlexicalized features&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;49.4&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;50.4&lt;/td&gt;
&lt;/tr&gt;

&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;Bowman et al. '15&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;+ Unigram and bigram features&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;99.7&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;&lt;em&gt;78.2&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;


&lt;tr class=&quot;section&quot;&gt;
&lt;th colspan=&quot;5&quot; style=&quot;background-color:transparent; color:#646464;&quot;&gt;Sentence vector-based models&lt;/th&gt;
&lt;/tr&gt;

&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;http://nlp.stanford.edu/pubs/snli_paper.pdf&quot;&gt;Bowman et al. '15&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;100D LSTM encoders&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;220k&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;84.8&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;77.6&lt;/td&gt;
&lt;/tr&gt;

&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;https://www.nyu.edu/projects/bowman/spinn.pdf&quot;&gt;Bowman et al. '16&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;300D LSTM encoders&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;3.0m&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;83.9&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;80.6&lt;/td&gt;
&lt;/tr&gt;

&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;http://arxiv.org/pdf/1511.06361v3.pdf&quot;&gt;Vendrov et al. '15&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;1024D GRU encoders w/ unsupervised 'skip-thoughts' pre-training&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;15m&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;98.8&lt;/td&gt;
&lt;td style=&quot;text-align: right&quot;&gt;81.4&lt;/td&gt;
&lt;/tr&gt;
...

</code></pre>
<p>I know I could use <code>requests</code> and <code>bs4.BeautifulSoup</code> to download and parse this page. But I could not figure out a way to extract URLs because there is not an easy to to pinpoint each individual row (there are other URLs outside of the table, so I could not say any URL extracted from the HTML is what I want).</p>
<p>Could anyone help me? Thank you in advance.</p>
<h1>Update</h1>
<p>The main difficulty is to extract URLs <strong>only</strong> from the leaderboard, which is tagged as</p>
<pre><code>&lt;h3&gt;Three-way classification&lt;/h3&gt;

&lt;blockquote&gt;
&lt;table class=&quot;newstuff&quot;&gt;
...

&lt;/table&gt;
&lt;/blockquote&gt;
</code></pre>
<p>Before and after this leaderboard, there are many contexts that is irrelevant to my purpose, where there are also a lot of URLs.</p>