title: Crawl data with Scrapy to load more?
body: <p>I need to crawl some news from this site: <a href="https://www.huxiu.com/channel/103.html" rel="nofollow noreferrer">https://www.huxiu.com/channel/103.html</a>. Here 103 is the news category id.        </p>
  
  <p>But I can only get the first page if not triggering the ajax to load more: 
  <a href="https://i.stack.imgur.com/mQawr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mQawr.png" alt="enter image description here"></a><br>
  Very strange the request url is the same for different news categories. </p>
  
  <p><a href="https://i.stack.imgur.com/HHm6y.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HHm6y.png" alt="enter image description here"></a></p>
  
  <p>And the page information was delivered by headers by the referer. And the page is sent by the form data.  </p>
  
  <p>Here is the snippet of my code:  </p>
  
  <pre><code>    self.page += 1
      url = "https://www.huxiu.com/channel/ajaxGetMore"
      method = "POST"
  
      headers = {
          "Host": "www.huxiu.com",
          "Origin": "https://www.huxiu.com",
          "Referer": "https://www.huxiu.com/channel/106.html",
          "User-Agent": (
              "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/"
              "537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Sa"
              "fari/537.36"
          ),
          "X-Requested-With": "XMLHttpRequest",
      }
  
      formdata = {
          "huxiu_hash_code": "9aee58d3507ecafed74df13e156ab01b",
          "page": str(self.page),
          "catId": "106"
      }
  
      yield FormRequest(
          url=url,
          method=method,
          headers=headers,
          formdata=formdata,
          callback=self.parse
      )
  </code></pre>
  
  <p>Which failed to load more news feeds. How can I send the post request to crawl more news?</p>