{"action":"created","issue":{"url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/174","repository_url":"https://api.github.com/repos/akhoury/nodebb-plugin-import","labels_url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/174/labels{/name}","comments_url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/174/comments","events_url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/174/events","html_url":"https://github.com/akhoury/nodebb-plugin-import/issues/174","id":136743716,"number":174,"title":"Data.eachOrphanedPost is very inefficient","user":{"login":"BenLubar","id":4257305,"avatar_url":"https://avatars.githubusercontent.com/u/4257305?v=3","gravatar_id":"","url":"https://api.github.com/users/BenLubar","html_url":"https://github.com/BenLubar","followers_url":"https://api.github.com/users/BenLubar/followers","following_url":"https://api.github.com/users/BenLubar/following{/other_user}","gists_url":"https://api.github.com/users/BenLubar/gists{/gist_id}","starred_url":"https://api.github.com/users/BenLubar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BenLubar/subscriptions","organizations_url":"https://api.github.com/users/BenLubar/orgs","repos_url":"https://api.github.com/users/BenLubar/repos","events_url":"https://api.github.com/users/BenLubar/events{/privacy}","received_events_url":"https://api.github.com/users/BenLubar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-26T16:24:38Z","updated_at":"2016-02-26T16:49:35Z","closed_at":null,"body":"The forum I'm importing has 813833 posts and is currently 529194 / 813833 on fixPostsToPidsStart.\r\n\r\n![CPU usage graph](https://cloud.githubusercontent.com/assets/4257305/13357740/6c02a9c6-dc71-11e5-9d2c-d0ac95efd9c4.png)\r\n\r\nHere are the last two lines of mongo's log file:\r\n\r\n```\r\n2016-02-26T16:14:05.504+0000 I COMMAND  [conn14] command 0.objects command: find { find: \"objects\", filter: { _key: \"_imported:_posts\", value: \"912903\" }, projection: { _id: 0, value: 1 }, limit: 1, batchSize: 1, singleBatch: true } planSummary: IXSCAN { _key: 1, value: -1 } keysExamined:1 docsExamined:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:141 locks:{ Global: { acquireCount: { r: 4 } }, MMAPV1Journal: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { R: 2 }, acquireWaitCount: { R: 2 }, timeAcquiringMicros: { R: 16955 } } } protocol:op_query 112ms\r\n2016-02-26T16:15:31.786+0000 I COMMAND  [conn20] command 0.objects command: find { find: \"objects\", filter: { _key: \"posts:pid\" }, sort: { score: 1 }, projection: { _id: 0, value: 1 }, skip: 601455, limit: 101 } planSummary: IXSCAN { _key: 1, score: -1 } keysExamined:601556 docsExamined:601556 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:6829 nreturned:101 reslen:2832 locks:{ Global: { acquireCount: { r: 13660 } }, MMAPV1Journal: { acquireCount: { r: 6832 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 61283 } }, Database: { acquireCount: { r: 6830 } }, Collection: { acquireCount: { R: 6830 }, acquireWaitCount: { R: 2 }, timeAcquiringMicros: { R: 164 } } } protocol:op_query 86148ms\r\n```\r\n\r\nI don't know much about MongoDB, but I do know that looking at hundreds of thousands of things spread across a file about an order of magnitude larger than the physical RAM of the machine is going to be very slow.\r\n\r\nIs there anything that can be done in this situation?"},"comment":{"url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/comments/189366909","html_url":"https://github.com/akhoury/nodebb-plugin-import/issues/174#issuecomment-189366909","issue_url":"https://api.github.com/repos/akhoury/nodebb-plugin-import/issues/174","id":189366909,"user":{"login":"akhoury","id":1398375,"avatar_url":"https://avatars.githubusercontent.com/u/1398375?v=3","gravatar_id":"","url":"https://api.github.com/users/akhoury","html_url":"https://github.com/akhoury","followers_url":"https://api.github.com/users/akhoury/followers","following_url":"https://api.github.com/users/akhoury/following{/other_user}","gists_url":"https://api.github.com/users/akhoury/gists{/gist_id}","starred_url":"https://api.github.com/users/akhoury/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akhoury/subscriptions","organizations_url":"https://api.github.com/users/akhoury/orgs","repos_url":"https://api.github.com/users/akhoury/repos","events_url":"https://api.github.com/users/akhoury/events{/privacy}","received_events_url":"https://api.github.com/users/akhoury/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T16:49:35Z","updated_at":"2016-02-26T16:49:35Z","body":"im gonna assume that the posts ordered all the time by timestamp, and avoid this whole fixPostsToPids phase. I will change it this weekend \r\n"}}