{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/2223","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/2223/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/2223/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/2223/events","html_url":"https://github.com/scrapy/scrapy/issues/2223","id":175698219,"number":2223,"title":"How to insert time-dependent parameter?","user":{"login":"caizixian","id":2891235,"avatar_url":"https://avatars.githubusercontent.com/u/2891235?v=3","gravatar_id":"","url":"https://api.github.com/users/caizixian","html_url":"https://github.com/caizixian","followers_url":"https://api.github.com/users/caizixian/followers","following_url":"https://api.github.com/users/caizixian/following{/other_user}","gists_url":"https://api.github.com/users/caizixian/gists{/gist_id}","starred_url":"https://api.github.com/users/caizixian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caizixian/subscriptions","organizations_url":"https://api.github.com/users/caizixian/orgs","repos_url":"https://api.github.com/users/caizixian/repos","events_url":"https://api.github.com/users/caizixian/events{/privacy}","received_events_url":"https://api.github.com/users/caizixian/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-08T08:48:38Z","updated_at":"2016-09-10T08:19:59Z","closed_at":"2016-09-09T17:14:57Z","body":"For example, if I have a spider like this\r\n```\r\nimport time\r\n\r\nclass Spider1(scrapy.Spider):\r\n    name = \"spider\"\r\n    allowed_domains = [\"example.com\"]\r\n    start_urls = [\"https://www.example.com/index.html\"]\r\n    def parse(self, response):\r\n        for page in range(1,5000000):\r\n            request = scrapy.Request(\"https://www.example.com/{}.html\".format(page),\r\n                                     method='POST',\r\n                                     body=json.dumps({\"time\":time.time()}))\r\n            yield request\r\n```\r\nAll the `time` in POST requests will be the time when the request is scheduled.\r\n\r\nWhat should I do to make the `time` be the actual time when the request is sent.\r\n\r\nTried to write middlewares to fill in the parameters, but it seems that the middleware methods are also called when the request is scheduled."},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/246099125","html_url":"https://github.com/scrapy/scrapy/issues/2223#issuecomment-246099125","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/2223","id":246099125,"user":{"login":"caizixian","id":2891235,"avatar_url":"https://avatars.githubusercontent.com/u/2891235?v=3","gravatar_id":"","url":"https://api.github.com/users/caizixian","html_url":"https://github.com/caizixian","followers_url":"https://api.github.com/users/caizixian/followers","following_url":"https://api.github.com/users/caizixian/following{/other_user}","gists_url":"https://api.github.com/users/caizixian/gists{/gist_id}","starred_url":"https://api.github.com/users/caizixian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caizixian/subscriptions","organizations_url":"https://api.github.com/users/caizixian/orgs","repos_url":"https://api.github.com/users/caizixian/repos","events_url":"https://api.github.com/users/caizixian/events{/privacy}","received_events_url":"https://api.github.com/users/caizixian/received_events","type":"User","site_admin":false},"created_at":"2016-09-10T08:19:59Z","updated_at":"2016-09-10T08:19:59Z","body":"Sorry for still commenting here.\r\n\r\nCould you please clarify this to me: how does scheduler decide when to put requests to downloader queue?\r\n\r\nFor instance, in the example above, it seems that tons of requests are scheduled in short intervals. Will these scheduled requests be added to the downloader queue immediately or be added to the queue when concurrent requests, let's say, are lower than `CONCURRENT_REQUESTS` . \r\n\r\nIn my own experiment, I wrote a downloader middleware which will insert some data into Redis database when the `process_request` is invoked. \r\n\r\nAnd I opened `redis-cli monitor` to monitor DB operations. Whether I call inserting when the request object is being constructed or the `process_request` is invoked, I can always see hundreds of db logs which is way more than download speed. "}}
{"actor":{"display_login":"caizixian"}}