{"action":"created","issue":{"url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/18","labels_url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/18/comments","events_url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/18/events","html_url":"https://github.com/piwik/plugin-QueuedTracking/issues/18","id":101658522,"number":18,"title":"Importing data with log importer queues all req in the same redis queue instead of distributing req across all queues","user":{"login":"jaceklubzinski","id":13852206,"avatar_url":"https://avatars.githubusercontent.com/u/13852206?v=3","gravatar_id":"","url":"https://api.github.com/users/jaceklubzinski","html_url":"https://github.com/jaceklubzinski","followers_url":"https://api.github.com/users/jaceklubzinski/followers","following_url":"https://api.github.com/users/jaceklubzinski/following{/other_user}","gists_url":"https://api.github.com/users/jaceklubzinski/gists{/gist_id}","starred_url":"https://api.github.com/users/jaceklubzinski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaceklubzinski/subscriptions","organizations_url":"https://api.github.com/users/jaceklubzinski/orgs","repos_url":"https://api.github.com/users/jaceklubzinski/repos","events_url":"https://api.github.com/users/jaceklubzinski/events{/privacy}","received_events_url":"https://api.github.com/users/jaceklubzinski/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/milestones/1","html_url":"https://github.com/piwik/plugin-QueuedTracking/milestones/Current%20sprint","labels_url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/milestones/1/labels","id":1159517,"number":1,"title":"Current sprint","description":null,"creator":{"login":"mattab","id":466765,"avatar_url":"https://avatars.githubusercontent.com/u/466765?v=3","gravatar_id":"","url":"https://api.github.com/users/mattab","html_url":"https://github.com/mattab","followers_url":"https://api.github.com/users/mattab/followers","following_url":"https://api.github.com/users/mattab/following{/other_user}","gists_url":"https://api.github.com/users/mattab/gists{/gist_id}","starred_url":"https://api.github.com/users/mattab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattab/subscriptions","organizations_url":"https://api.github.com/users/mattab/orgs","repos_url":"https://api.github.com/users/mattab/repos","events_url":"https://api.github.com/users/mattab/events{/privacy}","received_events_url":"https://api.github.com/users/mattab/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":10,"state":"open","created_at":"2015-06-11T01:45:54Z","updated_at":"2016-01-10T20:00:15Z","due_on":null,"closed_at":null},"comments":18,"created_at":"2015-08-18T13:44:35Z","updated_at":"2016-01-10T20:54:58Z","closed_at":null,"body":"Hello,\r\n\r\n* run import command: `/srv/piwik/current/misc/log-analytics/import_logs.py /srv/logs/$log --url=http://front.piwiktest --config=/srv/piwik/current/config/config.ini.php --recorders=2 --recorder-max-payload-size=300 --log-format-regex='(?P<ip>\\S+) \\S+ \\S+ \\[(?P<date>.*?) (?P<timezone>.*?)\\] \"\\S+ (?P<path>.*?) \\S+\" (?P<status>\\S+) (?P<length>\\S+) \"(?P<referrer>.*?)\" \"(?P<user_agent>.*?)\" (?P<host>[\\w\\-\\.]*)(?::\\d+)? \\S+ \\S+' --retry-max-attempts=100`\r\n\r\nWe import 150GB of logs\r\n\r\n* run this command `/srv/piwik/current/console --piwik-domain=http://front.piwiktest queuedtracking:monitor`\r\n\r\nWe got this output. You can see that almost all of the req are in one queue\r\n\r\n```\r\nUp to 8 workers will be used\r\n\r\nProcessor will start once there are at least 25 request sets in the queue\r\n\r\n205395 (22+4+14+205286+10+20+20+19) request sets left in queue. 8.01G used memory (9.82G peak). 1 workers active\r\n```\r\n\r\n* Expected to get more distributed count of req in each queue, so that processing of queue is much faster.\r\n"},"comment":{"url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/comments/170393230","html_url":"https://github.com/piwik/plugin-QueuedTracking/issues/18#issuecomment-170393230","issue_url":"https://api.github.com/repos/piwik/plugin-QueuedTracking/issues/18","id":170393230,"user":{"login":"tsteur","id":273120,"avatar_url":"https://avatars.githubusercontent.com/u/273120?v=3","gravatar_id":"","url":"https://api.github.com/users/tsteur","html_url":"https://github.com/tsteur","followers_url":"https://api.github.com/users/tsteur/followers","following_url":"https://api.github.com/users/tsteur/following{/other_user}","gists_url":"https://api.github.com/users/tsteur/gists{/gist_id}","starred_url":"https://api.github.com/users/tsteur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsteur/subscriptions","organizations_url":"https://api.github.com/users/tsteur/orgs","repos_url":"https://api.github.com/users/tsteur/repos","events_url":"https://api.github.com/users/tsteur/events{/privacy}","received_events_url":"https://api.github.com/users/tsteur/received_events","type":"User","site_admin":false},"created_at":"2016-01-10T20:54:58Z","updated_at":"2016-01-10T20:54:58Z","body":"@quba do you mind executing the command and posting the output? "}}