title: HTTPError: Forbidden
body: <p>I get <code>HTTPError: Forbidden</code> error when trying to read from a website that requires feedback response in the form of radio buttons. How to resolve this ???</p>
  
  <p>An example url link that causes the error is commented out:
  <code>url = 'http://www.londonstockexchange.com/exchange/news/market-news/market-news-detail/WWH/13673174.html'</code></p>
  
  <p>Here is the code:</p>
  
  <pre><code>import urllib.request as req
  import re
  
  from bs4 import BeautifulSoup
  
  
  # get all text from websites.
  url = 'https://www.bbc.co.uk/sport/football'
  #url = 'http://www.londonstockexchange.com/exchange/news/market-news/market-news-detail/WWH/13673174.html'
  html = req.urlopen(url).read()
  soup = BeautifulSoup(html)
  
  # kill all script and style elements
  for script in soup(["script", "style"]):
      script.extract()    # rip it out
  
  # get text
  text = soup.get_text()
  
  # break into lines and remove leading and trailing space on each
  lines = (line.strip() for line in text.splitlines())
  # break multi-headlines into a line each
  chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
  # drop blank lines
  text = '\n'.join(chunk for chunk in chunks if chunk)
  
  print(text)
  </code></pre>