{"action":"created","issue":{"url":"https://api.github.com/repos/mcollina/mosca/issues/428","repository_url":"https://api.github.com/repos/mcollina/mosca","labels_url":"https://api.github.com/repos/mcollina/mosca/issues/428/labels{/name}","comments_url":"https://api.github.com/repos/mcollina/mosca/issues/428/comments","events_url":"https://api.github.com/repos/mcollina/mosca/issues/428/events","html_url":"https://github.com/mcollina/mosca/issues/428","id":138226461,"number":428,"title":"Performance Issue in loading subscriptions [in qlobber.js via redis persistence]","user":{"login":"behrad","id":2151889,"avatar_url":"https://avatars.githubusercontent.com/u/2151889?v=3","gravatar_id":"","url":"https://api.github.com/users/behrad","html_url":"https://github.com/behrad","followers_url":"https://api.github.com/users/behrad/followers","following_url":"https://api.github.com/users/behrad/following{/other_user}","gists_url":"https://api.github.com/users/behrad/gists{/gist_id}","starred_url":"https://api.github.com/users/behrad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behrad/subscriptions","organizations_url":"https://api.github.com/users/behrad/orgs","repos_url":"https://api.github.com/users/behrad/repos","events_url":"https://api.github.com/users/behrad/events{/privacy}","received_events_url":"https://api.github.com/users/behrad/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2016-03-03T16:29:05Z","updated_at":"2016-03-05T09:46:22Z","closed_at":null,"body":"I have 67K subscribers with 5 topics subscribed each (around 335K subscriptions), when starting Mosca it takes minutes for it to start persistence on redis (to load subscriptions) and CPU is 100%\r\n\r\nI traced the code and found this line https://github.com/mcollina/mosca/blob/master/lib/persistence/redis.js#L108 is producing all the trouble. commenting it lets mosca to start in few seconds iterating all the keys via `keys ...*` command.\r\nSo it seems qlobber trie implementation is not scalable to many hundred thousands of additions.\r\n\r\nAsking @davedoesdev to think about performance improvements?\r\n\r\nChange qlobber with may be another trie/qlobber impl? I found some but not dig into them yet:\r\nhttps://github.com/isaacs/node-glob\r\nhttps://github.com/jeresig/trie-js\r\nhttps://github.com/rbranson/glob-trie.js?utm_source=javascriptweekly&utm_medium=email\r\nhttp://ejohn.org/blog/javascript-trie-performance-analysis/\r\n\r\nWriting a custom one?\r\n\r\nWhat do you suggest @mcollina ?"},"comment":{"url":"https://api.github.com/repos/mcollina/mosca/issues/comments/192614273","html_url":"https://github.com/mcollina/mosca/issues/428#issuecomment-192614273","issue_url":"https://api.github.com/repos/mcollina/mosca/issues/428","id":192614273,"user":{"login":"behrad","id":2151889,"avatar_url":"https://avatars.githubusercontent.com/u/2151889?v=3","gravatar_id":"","url":"https://api.github.com/users/behrad","html_url":"https://github.com/behrad","followers_url":"https://api.github.com/users/behrad/followers","following_url":"https://api.github.com/users/behrad/following{/other_user}","gists_url":"https://api.github.com/users/behrad/gists{/gist_id}","starred_url":"https://api.github.com/users/behrad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behrad/subscriptions","organizations_url":"https://api.github.com/users/behrad/orgs","repos_url":"https://api.github.com/users/behrad/repos","events_url":"https://api.github.com/users/behrad/events{/privacy}","received_events_url":"https://api.github.com/users/behrad/received_events","type":"User","site_admin":false},"created_at":"2016-03-05T09:46:22Z","updated_at":"2016-03-05T09:46:22Z","body":"On Node 4.3.1, will test with `return` added and let you know.\r\n\r\n\r\n\r\n> Am thinking if it will be difficult to write this iteratively.\r\n\r\nThat'd be great, I was thinking about asking you for an iterative version so that I can test against recursive one :+1: "}}