{"action":"created","issue":{"url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/53","repository_url":"https://api.github.com/repos/istresearch/scrapy-cluster","labels_url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/53/comments","events_url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/53/events","html_url":"https://github.com/istresearch/scrapy-cluster/issues/53","id":141229873,"number":53,"title":"_get_bin takes hours with queue size 1M.","user":{"login":"yrik","id":394729,"avatar_url":"https://avatars.githubusercontent.com/u/394729?v=3","gravatar_id":"","url":"https://api.github.com/users/yrik","html_url":"https://github.com/yrik","followers_url":"https://api.github.com/users/yrik/followers","following_url":"https://api.github.com/users/yrik/following{/other_user}","gists_url":"https://api.github.com/users/yrik/gists{/gist_id}","starred_url":"https://api.github.com/users/yrik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yrik/subscriptions","organizations_url":"https://api.github.com/users/yrik/orgs","repos_url":"https://api.github.com/users/yrik/repos","events_url":"https://api.github.com/users/yrik/events{/privacy}","received_events_url":"https://api.github.com/users/yrik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-03-16T10:24:02Z","updated_at":"2016-03-17T01:28:17Z","closed_at":null,"body":"I'm scraping etsy.com and queue size  become more than 1M. When I query for info/statistics it stuck on `_get_bin` function in `scrapy-cluster/redis-monitor/plugins/info_monitor.py` file. Also i takes 500MB memory for redis-monitor in that moment.\r\n\r\n1) What is the best way to keep queue size small?\r\n2) Perhaps `_get_bin` should be rewritten in more efficient way to calc statistics in the database."},"comment":{"url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/comments/197637586","html_url":"https://github.com/istresearch/scrapy-cluster/issues/53#issuecomment-197637586","issue_url":"https://api.github.com/repos/istresearch/scrapy-cluster/issues/53","id":197637586,"user":{"login":"madisonb","id":3504194,"avatar_url":"https://avatars.githubusercontent.com/u/3504194?v=3","gravatar_id":"","url":"https://api.github.com/users/madisonb","html_url":"https://github.com/madisonb","followers_url":"https://api.github.com/users/madisonb/followers","following_url":"https://api.github.com/users/madisonb/following{/other_user}","gists_url":"https://api.github.com/users/madisonb/gists{/gist_id}","starred_url":"https://api.github.com/users/madisonb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/madisonb/subscriptions","organizations_url":"https://api.github.com/users/madisonb/orgs","repos_url":"https://api.github.com/users/madisonb/repos","events_url":"https://api.github.com/users/madisonb/events{/privacy}","received_events_url":"https://api.github.com/users/madisonb/received_events","type":"User","site_admin":false},"created_at":"2016-03-17T01:28:17Z","updated_at":"2016-03-17T01:28:17Z","body":"For crawl depth of greater than 0, I like to think of the next layer deep as being exponential, like somewhere between ~ 10^x to 20^x, where x is the depth you are crawling. If there are 20 links per page, you are at 3.2million just by depth 3.\r\n\r\nThe link spider by default crawls all links at 1 layer, and decrements the next layer by a priority of 10. If you remove that decrement, all your links would be on the same priority level and I think Redis would just give you random items from that level, so your queue may not grow as big. Also, if you added 10 instead of subtracted 10, you would do a depth first crawl instead of a breadth first, which may be the best option for you to reduce your crawl queue size.\r\n\r\nRegardless, increasing the number of crawlers and IP addresses will help you burn through the queue faster - there is no way to circumvent the queue backlog, so you might as well crawl it as fast as you can. For each IP address you have, you can multiple your crawl rate by a factor of 1. 2 IPs = 2x the crawl rate, 3 IPs= 3x the crawl rate. This is all configurable for how you want it to be.\r\n\r\nLastly, there is a Queue Stat dump by the Redis Monitor, have you looked at that log file? That gives you the backlog of every domain queue for every spider, regardless of appid or crawlid. Issue #40 tracks this, as we dump it to the logs but is not part of the API yet. \r\n\r\nHope all this helps you figure out what will work best for your needs."}}