title: How to bypass AKAMAI bot detection for data scraping using requests_html, Beautiful Soup
body: <p>I am scraping data from the <a href="https://www.rakuten.co.jp/" rel="nofollow noreferrer">Rakuten</a> Japanese e-commerce website. I am using <a href="https://github.com/psf/requests-html" rel="nofollow noreferrer">requests-html</a> and <a href="https://www.crummy.com/software/BeautifulSoup/" rel="nofollow noreferrer">Beautiful soup</a>.
  And the problem is when I request from my local pc (127.0.0.1) it's working fine. But when I request from my ec2 server getting <code>Reference  #&lt;esi:vars name=&quot;AKAMAI_DEBUG_STRING&quot;/&gt;</code> this message and no data or HTML page is found. And another case when I use <code>wget</code> and request from the server the page URL I get a full page. But my script doesn't work.</p>
  <p><strong>Here is my code sample:</strong></p>
  <pre><code>from bs4 import BeautifulSoup
  from requests_html import HTMLSession
  
  def get_search_url(query_keyword):
      base_url = 'https://search.rakuten.co.jp/search/mall/'
      headers = {
          'upgrade-insecure-requests': '1',
          'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
          'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
          'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
      }
      session = HTMLSession()
      session.headers.update(headers)
  
      request_url = base_url + query_keyword
      resp = session.get(request_url)
      soup = BeautifulSoup(resp.text, &quot;lxml&quot;)
      return soup
  
  def feature_product_details(url):
      output_list = []
      for i in url.find_all(&quot;div&quot;, attrs={&quot;class&quot;: &quot;dui-card searchresultitem&quot;}):
          product_title = i.find(&quot;div&quot;, attrs={&quot;class&quot;: &quot;content title&quot;})
          if product_title is not None:
              product_title = product_title.getText()
          else:
              product_title = &quot;&quot;
  
          output = {
              'title': product_title,
          }
          output_list.append(output)
      print(output_list)
      return output_list
  
  def main_rakuten_product_search(query):
      query_keyword = query
      page = get_search_url(query_keyword)
      product_lists = feature_product_details(page)
      return product_lists
  
  if __name__ == '__main__':
      queries = '【レビュー特典中】スマートキー 電波遮断ケース 電波遮断ボックス リレーアタック防止用 キーケース '
      main_rakuten_product_search(queries)
  </code></pre>
  <p><strong>Sample output when running local server:</strong></p>
  <pre><code>[
      {
          &quot;title&quot;: &quot;【レビュー特典中】スマートキー 電波遮断ケース 電波遮断ボックス リレーアタック防止用 キーケース リレーアタック対策 ボックス 箱　電波遮断ケース RFIDブロッキング 高級PUレザー 高級車盗難防止 カーセキュリティ 高級感溢れ レクサス（グレー）&quot;,
      }
  ]
  </code></pre>
  <blockquote>
  <p>But don't get any response when running it on my server: Just show
  this message <code>Reference  #&lt;esi:vars name=&quot;AKAMAI_DEBUG_STRING&quot;/&gt;</code></p>
  </blockquote>
  <p>If anyone has any idea on how this could be done, I would be grateful to hear.</p>