title: Python request to crawl URL returns 404 Error while working inside the browser
body: <p>I have a crawling python script that hangs on a url: <code>pulsepoint.com/sellers.json</code></p>
  
  <p>The bot uses a standard request to get the content, but is returned Error 404. In the browser it works (there is a 301 redirect, but request can follow that). My first hunch is that this could be a request header issue, so I copied my browser configuration. The code looks like this</p>
  
  <pre><code>        crawled_url="pulsepoint.com"
          seller_json_url = 'http://{thehost}/sellers.json'.format(thehost=crawled_url)
          print(seller_json_url)
          myheaders = {
                  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0',
                  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                  'Accept-Language': 'fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3',
                  'Accept-Encoding': 'gzip, deflate, br',
                  'Connection': 'keep-alive',
                  'Pragma': 'no-cache',
                  'Cache-Control': 'no-cache'
              }
          r = requests.get(seller_json_url, headers=myheaders)
          logging.info("  %d" % r.status_code)
  </code></pre>
  
  <p>But I am still getting a 404 Error. </p>
  
  <p>My next guess:</p>
  
  <ul>
  <li>Login? Not used here</li>
  <li>Cookies? Not that I can see</li>
  </ul>
  
  <p>So how is their server blocking my bot? This is an url that is supposed to be crawled by the way, nothing illegal..</p>
  
  <p>Thanks in advance!</p>