title: scraping links from a website with a 403 error python
body: <p>I am trying to scrape the links from a list of links (all to different pages on the same website), but I keep running a 403 error. Here is an example of a link I am trying to scrape</p>
  
  <p><a href="https://www.spectatornews.com/page/6/?s=band" rel="nofollow noreferrer">https://www.spectatornews.com/page/6/?s=band</a></p>
  
  <p><a href="https://www.spectatornews.com/page/7/?s=band" rel="nofollow noreferrer">https://www.spectatornews.com/page/7/?s=band</a></p>
  
  <p>etc.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>getarticles = []
  
  from bs4 import BeautifulSoup
  import urllib.request
  
  for i in listoflinks:
      resp = urllib.request.urlopen(i)
      soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))
  
      for link in soup.find_all('a', href=True):
  
          getarticles.append(link['href'])
  </code></pre>
  
  <p>I've been trying to use some of the answers from <a href="https://stackoverflow.com/questions/16627227/http-error-403-in-python-3-web-scraping">HTTP error 403 in Python 3 Web Scraping</a> but I haven't had much success. I'm not sure if I'm applying them to my whole list of links correctly. I attempted to use one of the solutions below by using a header, but that returns a HTTP 406 error: not acceptable</p>
  
  <p>Here is my code attempted to fix:</p>
  
  <pre><code>getarticles = []
  from bs4 import BeautifulSoup
  
  from bs4 import BeautifulSoup
  import urllib.request
  
  for i in listoflinks:
      req=urllib.request.Request(i, headers={'User-Agent': 'Mozilla/5.0'})
      resp = urllib.request.urlopen(req)
      soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))
  
      for link in soup.find_all('a', href=True):
  
          getarticles.append(link['href'])
  </code></pre>
  
  <p>Any help is greatly appreciated. I'm very new to this, so as much as you can explain and help is great. I'd just like to collect the links from my list of websites!</p>
  
  <p>Thank you</p>