title: Website blocks Python crawler. Searching for Idea to avoid
body: <p>I want to crawl data from Object-sites from <a href="https://www.fewo-direkt.de" rel="nofollow noreferrer">https://www.fewo-direkt.de</a> (in US <a href="https://www.homeaway.com/" rel="nofollow noreferrer">https://www.homeaway.com/</a>) like this: <a href="https://www.fewo-direkt.de/ferienwohnung-ferienhaus/p8735326" rel="nofollow noreferrer">https://www.fewo-direkt.de/ferienwohnung-ferienhaus/p8735326</a>
  But if the crawler tries to launch the page I'll get only a page with the code below. I think fewo blocks crawler, but I don't know how and wheter there is a pssible way to avoid. Have anyone an idea?</p>
  
  <p>Python, requests, BeautifulSoup - With other Websites it works fine.</p>
  
  <pre><code>&lt;html style="height:100%"&gt;
     &lt;head&gt;
        &lt;meta content="NOINDEX, NOFOLLOW" name="ROBOTS"/&gt;
        &lt;meta content="telephone=no" name="format-detection"/&gt;
        &lt;meta content="initial-scale=1.0" name="viewport"/&gt;
        &lt;meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible"/&gt;
        &lt;script src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3" type="text/javascript"&gt;&lt;/script&gt;
     &lt;/head&gt;
     &lt;body style="margin:0px;height:100%"&gt;&lt;iframe frameborder="0" height="100%" marginheight="0px" marginwidth="0px" src="/_Incapsula_Resource?CWUDNSAI=20&amp;amp;xinfo=5-259174360-0%200NNN%20RT%281546012021046%20144%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B15%283%2c901868%2c0%29%20U5&amp;amp;incident_id=877000750783982903-1038286134589588661&amp;amp;edet=15&amp;amp;cinfo=03000000" width="100%"&gt;Request unsuccessful. Incapsula incident ID: 877000750783982903-1038286134589588661&lt;/iframe&gt;&lt;/body&gt;
  &lt;/html&gt;
  </code></pre>