<p>NYTimes has changed it's internal html structure since 2014. Newspaper3K will work fine if you try to parse articles published before 2014.</p>

<p>Other things to take into account: </p>

<ul>
<li>1980 articles are not available. </li>
<li>Articles before 1970 are not digitized (except 1964).</li>
<li>1970-1979 articles have lots of words splitted in the middle by a space.</li>
<li>If you parse with Newspaper3k several articles will contain only "NYTimes.com no longer supports Internet Explorer 9 or earlier. Please upgrade your browser."</li>
<li>Lot of articles will have the following texts inserted in the middle:</li>
</ul>

<p>"\n\nNewsletter Sign Up Continue reading the main story Sign Up for the Opinion Today Newsletter Every weekday, get thought-provoking commentary from Op-Ed columnists, the Times editorial board and contributing writers from around the world. Please verify you're not a robot by clicking the box. Invalid email address. Please re-enter. You must select a newsletter to subscribe to. Sign Up You will receive emails containing news content , updates and promotions from The New York Times. You may opt-out at any time. You agree to receive occasional updates and special offers for The New York Times's products and services. Thank you for subscribing. An error has occurred. Please try again later. View all New York Times newsletters.\n\n"</p>

<p>"\n\nNewsletter Sign Up Continue reading the main story Please verify you're not a robot by clicking the box. Invalid email address. Please re-enter. You must select a newsletter to subscribe to. Sign Up You will receive emails containing news content , updates and promotions from The New York Times. You may opt-out at any time. You agree to receive occasional updates and special offers for The New York Times's products and services. Thank you for subscribing. An error has occurred. Please try again later. View all New York Times newsletters.\n"</p>

<ul>
<li>Most blogs (blogs appear in 2010) will have also undesired texts inserted.</li>
</ul>

<p>If you are ok with data from 1990 to 2016 check the dataset used in this paper: <a href="https://arxiv.org/abs/1703.00607" rel="nofollow noreferrer">https://arxiv.org/abs/1703.00607</a> it's available online.</p>

<p>In case you need newer articles I thing you should write your own parser. I'm working on it but I didn't finished yet.</p>
Library: Newspaper (Newspaper3k) Trying to Parse Links From Main Page of News (Source) Site
<p>I am trying to parse from a set of links generated by using the python library called <a href="https://github.com/codelucas/newspaper" rel="nofollow noreferrer">Newspaper</a></p>

<h1>Goal:</h1>

<p>To parse every link from the main page (or specific page such as category) of a news site.</p>

<h1>Problem:</h1>

<ol>
<li>I generate an AttributeError when attempting to pass an 'article_link' into the 'Article()' method.</li>
<li>Using separate code to parse a single link from 'The New York Times', the text printed does not print the whole article.</li>
</ol>

<h1>Code Producing Problem 1:</h1>

<pre><code>import newspaper
from newspaper import Article

nyt_paper = newspaper.build(
    'http://nytimes.com/section/todayspaper', memoize_articles=False)
print(nyt_paper.size())

processed_link_list = []
for article_link in nyt_paper.articles:
    article = Article(url=article_link)
    article.download()
    article.html
    article.parse()
    print(article.authors)
    processed_link_list.append(article_link)

if len(nyt_paper.size()) is len(processed_link_list):
    print('All Links Processed')
else:
    print('All Links **NOT** Processed')
</code></pre>

<h1>Error Output:</h1>

<pre><code>Traceback (most recent call last):
  File "nyt_today.py", line 31, in &lt;module&gt;
    article = Article(url=article_link)
  File "C:\...\lib\site-packages\newspaper\article.py", line 60, in __init__
    scheme = urls.get_scheme(url)
  File "C:\...\lib\site-packages\newspaper\urls.py", line 279, in get_scheme
    return urlparse(abs_url, **kwargs).scheme
  File "C:\...\lib\urllib\parse.py", line 367, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "C:\...\lib\urllib\parse.py", line 123, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "C:\...\lib\urllib\parse.py", line 107, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "C:\...\lib\urllib\parse.py", line 107, in &lt;genexpr&gt;
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'Article' object has no attribute 'decode'
</code></pre>

<h1>Code Producing Problem 2:</h1>

<pre><code>from newspaper import Article
from newspaper import fulltext
import requests

nyt_url = 'https://www.nytimes.com/2019/02/26/opinion/trump-kim-vietnam.html'
article = Article(nyt_url)
article.download()
print(article.html)
article.parse()
print(article.authors)
print(article.text)
</code></pre>

<p>I have also tried this 'fulltext' method exampled in the documentation to print the text:</p>

<pre><code>article_html = requests.get(nyt_url).text
full_text = fulltext(article_html)
print(full_text)
</code></pre>

<p>However, although the <em>Entire</em> article text is ouput to the</p>

<pre><code>print(article.html)
</code></pre>

<p>the </p>

<pre><code>print(article.text)
</code></pre>

<p>does not print it all.  The original link, HTML Output and Printed Text Output can be seen below:</p>

<p>Link: <a href="https://www.nytimes.com/2019/02/26/opinion/trump-kim-vietnam.html" rel="nofollow noreferrer">https://www.nytimes.com/2019/02/26/opinion/trump-kim-vietnam.html</a> </p>

<p>Html Output: <a href="https://pastebin.com/A5tFdKap" rel="nofollow noreferrer">see this pastebin for truncated output</a></p>

<p>Printed text: <a href="https://pastebin.com/sHiyb3XU" rel="nofollow noreferrer">see this printed text does not print the entire article</a></p>

<p>Any help would be much appreciated.</p>