{"action":"created","issue":{"url":"https://api.github.com/repos/Osterjour/line-by-line/issues/3","repository_url":"https://api.github.com/repos/Osterjour/line-by-line","labels_url":"https://api.github.com/repos/Osterjour/line-by-line/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/Osterjour/line-by-line/issues/3/comments","events_url":"https://api.github.com/repos/Osterjour/line-by-line/issues/3/events","html_url":"https://github.com/Osterjour/line-by-line/issues/3","id":118324128,"number":3,"title":"After about 1.4M lines I hit memory issues","user":{"login":"OmgImAlexis","id":6525926,"avatar_url":"https://avatars.githubusercontent.com/u/6525926?v=3","gravatar_id":"","url":"https://api.github.com/users/OmgImAlexis","html_url":"https://github.com/OmgImAlexis","followers_url":"https://api.github.com/users/OmgImAlexis/followers","following_url":"https://api.github.com/users/OmgImAlexis/following{/other_user}","gists_url":"https://api.github.com/users/OmgImAlexis/gists{/gist_id}","starred_url":"https://api.github.com/users/OmgImAlexis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OmgImAlexis/subscriptions","organizations_url":"https://api.github.com/users/OmgImAlexis/orgs","repos_url":"https://api.github.com/users/OmgImAlexis/repos","events_url":"https://api.github.com/users/OmgImAlexis/events{/privacy}","received_events_url":"https://api.github.com/users/OmgImAlexis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-23T05:55:48Z","updated_at":"2016-05-23T04:40:25Z","closed_at":null,"body":"Even though this library claims it doesn't load the file into memory there is either a memory leak or it **IS** loading the file into memory. I'm trying to parse a 7M+ line file going line by line and then inserting it into a mongdb database. I tried this a few times and each time nodejs exited and told me there was a memory issue.\r\n\r\nI ended up doing something like this without the library and I'm not getting a single memory issue.\r\nWhen I used my method I was getting at most 1MB of extra RAM being used compared to IDLE and I'm pretty sure that's just the mongodb connection.\r\nUsing your library I was hitting 130MB - 200MB of RAM. When I first open node with my project I sit on about 70MB of RAM on IDLE.\r\n\r\n````\r\nvar fs = require('fs'),\r\n    es = require(\"event-stream\");\r\n\r\nvar lineNumber = 1;\r\n\r\nvar s = fs.createReadStream('import.txt').pipe(es.split()).pipe(es.mapSync(function(line){\r\n    // pause the readstream\r\n    s.pause();\r\n\r\n    lineNumber += 1;\r\n\r\n    (function(){\r\n        // Run my line by line code here\r\n        // resume the readstream\r\n        s.resume();\r\n    });\r\n}).on('error', function(){\r\n    console.log('Error while reading file.');\r\n}).on('end', function(){\r\n    console.log('Read entirefile.');\r\n}));\r\n````"},"comment":{"url":"https://api.github.com/repos/Osterjour/line-by-line/issues/comments/220886086","html_url":"https://github.com/Osterjour/line-by-line/issues/3#issuecomment-220886086","issue_url":"https://api.github.com/repos/Osterjour/line-by-line/issues/3","id":220886086,"user":{"login":"trvbr","id":6646266,"avatar_url":"https://avatars.githubusercontent.com/u/6646266?v=3","gravatar_id":"","url":"https://api.github.com/users/trvbr","html_url":"https://github.com/trvbr","followers_url":"https://api.github.com/users/trvbr/followers","following_url":"https://api.github.com/users/trvbr/following{/other_user}","gists_url":"https://api.github.com/users/trvbr/gists{/gist_id}","starred_url":"https://api.github.com/users/trvbr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trvbr/subscriptions","organizations_url":"https://api.github.com/users/trvbr/orgs","repos_url":"https://api.github.com/users/trvbr/repos","events_url":"https://api.github.com/users/trvbr/events{/privacy}","received_events_url":"https://api.github.com/users/trvbr/received_events","type":"User","site_admin":false},"created_at":"2016-05-23T04:40:25Z","updated_at":"2016-05-23T04:40:25Z","body":"@ralfschiffert Curious, did removing pause/resume fix it?"}}
{"actor":{"display_login":"trvbr"}}