{"action":"created","issue":{"url":"https://api.github.com/repos/louischatriot/nedb/issues/472","repository_url":"https://api.github.com/repos/louischatriot/nedb","labels_url":"https://api.github.com/repos/louischatriot/nedb/issues/472/labels{/name}","comments_url":"https://api.github.com/repos/louischatriot/nedb/issues/472/comments","events_url":"https://api.github.com/repos/louischatriot/nedb/issues/472/events","html_url":"https://github.com/louischatriot/nedb/issues/472","id":192021855,"number":472,"title":"Memory usage seems unexpectedly high (RAM usage is 10x disk usage)","user":{"login":"jpx","id":1085293,"avatar_url":"https://avatars.githubusercontent.com/u/1085293?v=3","gravatar_id":"","url":"https://api.github.com/users/jpx","html_url":"https://github.com/jpx","followers_url":"https://api.github.com/users/jpx/followers","following_url":"https://api.github.com/users/jpx/following{/other_user}","gists_url":"https://api.github.com/users/jpx/gists{/gist_id}","starred_url":"https://api.github.com/users/jpx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpx/subscriptions","organizations_url":"https://api.github.com/users/jpx/orgs","repos_url":"https://api.github.com/users/jpx/repos","events_url":"https://api.github.com/users/jpx/events{/privacy}","received_events_url":"https://api.github.com/users/jpx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-28T14:37:27Z","updated_at":"2016-11-29T08:18:36Z","closed_at":null,"body":"Hi,\r\n\r\nWe are facing high memory usage when a collection reaches thousands of documents.\r\nUsed memory seems to be consistently 10x the disk storage used by .db.\r\n\r\nAre you usually facing similar figures ?\r\n\r\nDisabling indexing makes no noticeable difference.\r\nThe documents are batch-inserted.\r\nMemory usage does not change after reloading the database (compared to run where the insertion took place).\r\n\r\nTesting platform: Windows 10, nodejs v6.9.1, npm v3.10.8.\r\nAlso reproduced under Android (6.0), with nodejs running in a thread.\r\n\r\nWe measured following RAM usage stats, for\r\n3000 documents: 10MB, with a .db of 0,97MB\r\n20000 documents: 57MB, with a .db of 6,5MB\r\n30000 documents: 95MB, with a .db of 9,8MB\r\n40000 documents: 135MB with a .db of 13,3MB\r\n60000 documents: 188MB with a .db of 19,5MB\r\n"},"comment":{"url":"https://api.github.com/repos/louischatriot/nedb/issues/comments/263504993","html_url":"https://github.com/louischatriot/nedb/issues/472#issuecomment-263504993","issue_url":"https://api.github.com/repos/louischatriot/nedb/issues/472","id":263504993,"user":{"login":"jampy","id":820758,"avatar_url":"https://avatars.githubusercontent.com/u/820758?v=3","gravatar_id":"","url":"https://api.github.com/users/jampy","html_url":"https://github.com/jampy","followers_url":"https://api.github.com/users/jampy/followers","following_url":"https://api.github.com/users/jampy/following{/other_user}","gists_url":"https://api.github.com/users/jampy/gists{/gist_id}","starred_url":"https://api.github.com/users/jampy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jampy/subscriptions","organizations_url":"https://api.github.com/users/jampy/orgs","repos_url":"https://api.github.com/users/jampy/repos","events_url":"https://api.github.com/users/jampy/events{/privacy}","received_events_url":"https://api.github.com/users/jampy/received_events","type":"User","site_admin":false},"created_at":"2016-11-29T08:18:36Z","updated_at":"2016-11-29T08:18:36Z","body":"When loaded, the collection is fully kept in memory as \"*real*\" JavaScript objects, not strings.\r\n\r\nI *guess* there is some memory overhead for handling each single *value* contained within - by the JavaScript engine itself.\r\n\r\nThis will likely also depend on the kind of values.\r\n\r\nA collection where each document holds a single, but very long string will probably need few extra memory, whereas a document holding lots of sub-documents or numbers will probably need more..."}}
{"actor":{"display_login":"jampy"}}