{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/3160","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/3160/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/3160/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/3160/events","html_url":"https://github.com/scrapy/scrapy/pull/3160","id":303688530,"node_id":"MDExOlB1bGxSZXF1ZXN0MTczODkzODY3","number":3160,"title":"Adding domain","user":{"login":"tianhuil","id":1906321,"node_id":"MDQ6VXNlcjE5MDYzMjE=","avatar_url":"https://avatars0.githubusercontent.com/u/1906321?v=4","gravatar_id":"","url":"https://api.github.com/users/tianhuil","html_url":"https://github.com/tianhuil","followers_url":"https://api.github.com/users/tianhuil/followers","following_url":"https://api.github.com/users/tianhuil/following{/other_user}","gists_url":"https://api.github.com/users/tianhuil/gists{/gist_id}","starred_url":"https://api.github.com/users/tianhuil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tianhuil/subscriptions","organizations_url":"https://api.github.com/users/tianhuil/orgs","repos_url":"https://api.github.com/users/tianhuil/repos","events_url":"https://api.github.com/users/tianhuil/events{/privacy}","received_events_url":"https://api.github.com/users/tianhuil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-03-09T01:04:09Z","updated_at":"2018-09-03T10:14:07Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/3160","html_url":"https://github.com/scrapy/scrapy/pull/3160","diff_url":"https://github.com/scrapy/scrapy/pull/3160.diff","patch_url":"https://github.com/scrapy/scrapy/pull/3160.patch"},"body":"Merges DomainScheduler implemented in https://github.com/tianhuil/domain_scheduler into scrapy. It scrapes in a domain-smart way: by round-robin cycling through the domains. This has two benefits:\r\n\r\n1. Spreading out load on the target servers instead of hitting the server with many requests at once\r\n2. Reducing delays caused by server-throttling or scrapy's own `CONCURRENT_REQUESTS_PER_IP` restrictions.  Empirical testing has shown this to be quite effective.\r\n\r\nIt implements the solution proposed in https://github.com/scrapy/scrapy/issues/1802#issue-135260562 which found similar performance improvements.  **Original proposal** was first posted https://github.com/scrapy/scrapy/issues/1802 (code no longer available), https://github.com/scrapy/scrapy/issues/2474, and https://github.com/scrapy/scrapy/issues/3140.  \r\n\r\n**Note:** It requires more than just using `SCHEDULER_PRIORITY_QUEUE` as it needs an API change to the queue (passing in a non-integer key, i.e. the domain, to a new round robin queue).\r\n\r\n- Implements https://github.com/scrapy/scrapy/issues/3140\r\n- Depends on merged queuelib PR https://github.com/scrapy/queuelib/pull/21\r\n\r\nWould love to get feedback @pablohoffman, @dangra, and @kmike!"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/418069275","html_url":"https://github.com/scrapy/scrapy/pull/3160#issuecomment-418069275","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/3160","id":418069275,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODA2OTI3NQ==","user":{"login":"kmike","id":107893,"node_id":"MDQ6VXNlcjEwNzg5Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/107893?v=4","gravatar_id":"","url":"https://api.github.com/users/kmike","html_url":"https://github.com/kmike","followers_url":"https://api.github.com/users/kmike/followers","following_url":"https://api.github.com/users/kmike/following{/other_user}","gists_url":"https://api.github.com/users/kmike/gists{/gist_id}","starred_url":"https://api.github.com/users/kmike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmike/subscriptions","organizations_url":"https://api.github.com/users/kmike/orgs","repos_url":"https://api.github.com/users/kmike/repos","events_url":"https://api.github.com/users/kmike/events{/privacy}","received_events_url":"https://api.github.com/users/kmike/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T10:14:07Z","updated_at":"2018-09-03T10:14:07Z","author_association":"MEMBER","body":"hey @tianhuil, I've left some more comments. Besides them, there are the following issues:\r\n\r\n1. Merge conflicts should be resolved.\r\n2. Tests should pass.\r\n3. It is still not documented that request priorities don't work. I'd prefer them working, but if they don't, it should be clearly documented, both in scheduler docs and in Request docs."}}
{"actor":{"display_login":"kmike"}}