{"action":"created","issue":{"url":"https://api.github.com/repos/binux/pyspider/issues/483","repository_url":"https://api.github.com/repos/binux/pyspider","labels_url":"https://api.github.com/repos/binux/pyspider/issues/483/labels{/name}","comments_url":"https://api.github.com/repos/binux/pyspider/issues/483/comments","events_url":"https://api.github.com/repos/binux/pyspider/issues/483/events","html_url":"https://github.com/binux/pyspider/issues/483","id":163805496,"number":483,"title":"Get same result with different page number","user":{"login":"AndyZYZ","id":6388391,"avatar_url":"https://avatars.githubusercontent.com/u/6388391?v=3","gravatar_id":"","url":"https://api.github.com/users/AndyZYZ","html_url":"https://github.com/AndyZYZ","followers_url":"https://api.github.com/users/AndyZYZ/followers","following_url":"https://api.github.com/users/AndyZYZ/following{/other_user}","gists_url":"https://api.github.com/users/AndyZYZ/gists{/gist_id}","starred_url":"https://api.github.com/users/AndyZYZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndyZYZ/subscriptions","organizations_url":"https://api.github.com/users/AndyZYZ/orgs","repos_url":"https://api.github.com/users/AndyZYZ/repos","events_url":"https://api.github.com/users/AndyZYZ/events{/privacy}","received_events_url":"https://api.github.com/users/AndyZYZ/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-05T09:15:26Z","updated_at":"2016-07-08T11:05:43Z","closed_at":null,"body":"Hi\r\n\r\nI'm trying to crawl a list from a online video website. \r\n\r\nHere is the url\r\nhttp://www.bilibili.com/newlist.html?page=1\r\nhttp://www.bilibili.com/newlist.html?page=2\r\n...\r\n\r\nIt's really weird that while crawling every page I get the same result as page 1.\r\n\r\nCode:\r\n````\r\n#!/usr/bin/env python\r\n# -*- encoding: utf-8 -*-\r\n# Created on 2016-07-05 14:45:23\r\n# Project: bilinew\r\n\r\nfrom pyspider.libs.base_handler import *\r\nimport re\r\n\r\nclass Handler(BaseHandler):\r\n    crawl_config = {\r\n    }\r\n\r\n    def __init__(self):\r\n        self.base_url = 'http://www.bilibili.com/newlist.html?page='\r\n        self.page_num = 1\r\n        self.total_num = 30\r\n    \r\n    @every(minutes=24 * 60)\r\n    def on_start(self):\r\n        while self.page_num <= self.total_num:\r\n            url = self.base_url + str(self.page_num)\r\n            #print url\r\n            self.crawl(url, callback=self.index_page)\r\n            self.page_num += 1\r\n\r\n    @config(age=10*24*60*60)\r\n    def index_page(self, response):\r\n        for each in response.doc('a[href^=\"http\"]').items():\r\n            if re.match(\"http://www.bilibili.com/video/av\\d{7}\", each.attr.href):\r\n                self.crawl(each.attr.href, callback=self.detail_page)\r\n        #self.crawl(response.doc('.nextPage').attr.href, callback=self.index_page)\r\n\r\n    @config(priority=2)\r\n    def detail_page(self, response):\r\n        return {\r\n            \"url\": response.url,\r\n            \"title\": response.doc('title').text(),\r\n        }\r\n````\r\n\r\nAfter digging deeper I find after I open one of the urls in my browser it can be crawled properly.\r\n\r\nAny one can help me?\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/binux/pyspider/issues/comments/231333915","html_url":"https://github.com/binux/pyspider/issues/483#issuecomment-231333915","issue_url":"https://api.github.com/repos/binux/pyspider/issues/483","id":231333915,"user":{"login":"binux","id":646451,"avatar_url":"https://avatars.githubusercontent.com/u/646451?v=3","gravatar_id":"","url":"https://api.github.com/users/binux","html_url":"https://github.com/binux","followers_url":"https://api.github.com/users/binux/followers","following_url":"https://api.github.com/users/binux/following{/other_user}","gists_url":"https://api.github.com/users/binux/gists{/gist_id}","starred_url":"https://api.github.com/users/binux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binux/subscriptions","organizations_url":"https://api.github.com/users/binux/orgs","repos_url":"https://api.github.com/users/binux/repos","events_url":"https://api.github.com/users/binux/events{/privacy}","received_events_url":"https://api.github.com/users/binux/received_events","type":"User","site_admin":false},"created_at":"2016-07-08T11:05:43Z","updated_at":"2016-07-08T11:05:43Z","body":"No, click next page doesn't work for me. Did you login?"}}
{"actor":{"display_login":"binux"}}