title: Scrapy can not crawl data down even through set the User-Agent, what's the reason?
body: <p>I am studying Scrapy, I want to scrapy <a href="https://careers.tencent.com/search.html" rel="nofollow noreferrer">this site</a>.  </p>
  
  <p>In my spider:</p>
  
  <pre><code>import scrapy
  
  class TencentHrSpider(scrapy.Spider):
      name = 'tencent_hr'
      allowed_domains = ['careers.tencent.com']
      start_urls = ['http://careers.tencent.com/search.html']
  
      def parse(self, response):
  
          div_list = response.xpath('//div[@class="recruit-list"]')
  
          print(div_list)  # there get `[]`, no data in it.
  </code></pre>
  
  <p>When I start crawl, there is no data output. 
  why? </p>
  
  <p>I have set the request header User-Agent in <code>settings.py</code>:</p>
  
  <pre><code>USER_AGENT_LIST=[
  'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
      "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
      "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
      "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
      "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
      "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
  ]
  import random
  USER_AGENT = random.choice(USER_AGENT_LIST)
  </code></pre>
  
  <hr>
  
  <p><strong>Edit - 01</strong></p>
  
  <p>Is it possible to find the reason? any error log to trace?</p>
  
  <hr>
  
  <p><strong>EDIT -02</strong></p>
  
  <p>Why if the data is request from a API by AJAX, the Scrapy can not get data? we know it can down load the whole page, is it possible to run the script like browser?</p>