title: Cannot scrape web page using Python
body: <p>I have posted a similar <a href="https://stackoverflow.com/questions/42472840/cannot-access-the-same-web-page-while-scrapping-using-python?noredirect=1#comment72086220_42472840">question</a> before. I was trying to scrape a <a href="https://www.zameen.com/&#39;" rel="nofollow noreferrer">web page</a> using the following approach</p>
  
  <pre><code>import requests
  
  url = 'https://www.zameen.com/'
  res = requests.get(url)
  data = res.text
  print(data)
  </code></pre>
  
  <p>Its response says I am either a BOT or Javascript is not enabled. So, I have check but Javascript is enabled. So I tried another approach of using fake user agent with the following code</p>
  
  <pre><code>from fake_useragent import UserAgent
  headers = {}
  headers['User-Agent'] = str(ua.chrome)
  web_page = requests.get(url,headers=headers)
  print(web_page.content)
  </code></pre>
  
  <p>Response:</p>
  
  <pre><code>b'&lt;!DOCTYPE html&gt;\n\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\n\t\n\t\n\t\n\n&lt;head&gt;\n&lt;META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"&gt;\n&lt;meta http-equiv="cache-control" content="max-age=0" /&gt;\n&lt;meta http-equiv="cache-control" content="no-cache" /&gt;\n&lt;meta http-equiv="expires" content="0" /&gt;\n&lt;meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" /&gt;\n&lt;meta http-equiv="pragma" content="no-cache" /&gt;\n&lt;meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?Ref=/&amp;amp;distil_RID=053235A2-0030-11E7-8429-B03805AB611E&amp;amp;distil_TID=20170303163950" /&gt;\n&lt;script type="text/javascript"&gt;\n\t(function(window){\n\t\ttry {\n\t\t\tif (typeof sessionStorage !== \'undefined\'){\n\t\t\t\tsessionStorage.setItem(\'distil_referrer\', document.referrer);\n\t\t\t}\n\t\t} catch (e){}\n\t})(window);\n&lt;/script&gt;\n&lt;script type="text/javascript" src="/ga368490.js" defer&gt;&lt;/script&gt;&lt;style type="text/css"&gt;#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#caexxxzxycbzutyvy{display:none!important}&lt;/style&gt;&lt;/head&gt;\n&lt;body&gt;\n&lt;div id="distil_ident_block"&gt;&amp;nbsp;&lt;/div&gt;\n&lt;/body&gt;\n&lt;/html&gt;\n'
  </code></pre>
  
  <p>It again detected me as a robot. So I checked whether I can fetch the data from the website or not. Then I used robotparser from urllib</p>
  
  <pre><code>from urllib import robotparser
  
  req = robotparser.RobotFileParser()
  req.set_url(url)
  req.read()
  print(req.can_fetch('*','https://www.zameen.com/'))
  </code></pre>
  
  <p>Returns:</p>
  
  <pre><code>TRUE # Means I can fetch the data from the website. 
  </code></pre>
  
  <p>Is there any way to get the data from this web page ? Thanks</p>