title: Web-scraping articles from WSJ using Beautifulsoup in python 3.7?
body: <p>I am trying to scrape articles from the Wall Street Journal using Beautifulsoup in Python. However, the code which I am running is executing without any error (exit code 0) but no results. I don't understand what is happening? Why this code is not giving expected results.</p>
  <p>I even have paid a subscription.</p>
  <p>I know that something is not right but I can't locate the problem.</p>
  <pre><code>import time
  
  import requests
  
  from bs4 import BeautifulSoup
  
  url = 'https://www.wsj.com/search/term.html?KEYWORDS=cybersecurity&amp;min-date=2018/04/01&amp;max-date=2019/03/31' \
    '&amp;isAdvanced=true&amp;daysback=90d&amp;andor=AND&amp;sort=date-desc&amp;source=wsjarticle,wsjpro&amp;page={}'
  
  pages = 32
  for page in range(1, pages+1):
      res = requests.get(url.format(page))
      soup = BeautifulSoup(res.text,&quot;lxml&quot;)
      for item in soup.select(&quot;.items.hedSumm li &gt; a&quot;):
          resp = requests.get(item.get(&quot;href&quot;))
          _href = item.get(&quot;href&quot;)
  
          try:
              resp = requests.get(_href)
          except Exception as e:
              try:
              resp = requests.get(&quot;https://www.wsj.com&quot; + _href)
          except Exception as e:
              continue
      sauce = BeautifulSoup(resp.text,&quot;lxml&quot;)
      date = sauce.select(&quot;time.timestamp.article__timestamp.flexbox__flex--1&quot;)
      date = date[0].text
      tag = sauce.select(&quot;li.article-breadCrumb span&quot;).text
      title = sauce.select_one(&quot;h1.wsj-article-headline&quot;).text
      content = [elem.text for elem in sauce.select(&quot;p.article-content&quot;)]
      print(f'{date}\n {tag}\n {title}\n {content}\n')
  
      time.sleep(3)
  </code></pre>
  <p>As I wrote in the code, I am trying to scrape date, title, tag, and content of all the articles. It would be helpful if I can get suggestions about my mistakes, what should I do to get the desired results.</p>