{"action":"created","issue":{"url":"https://api.github.com/repos/nodejs/node/issues/8955","repository_url":"https://api.github.com/repos/nodejs/node","labels_url":"https://api.github.com/repos/nodejs/node/issues/8955/labels{/name}","comments_url":"https://api.github.com/repos/nodejs/node/issues/8955/comments","events_url":"https://api.github.com/repos/nodejs/node/issues/8955/events","html_url":"https://github.com/nodejs/node/issues/8955","id":181360507,"number":8955,"title":"Streams memory leak","user":{"login":"devmetal","id":3758485,"avatar_url":"https://avatars.githubusercontent.com/u/3758485?v=3","gravatar_id":"","url":"https://api.github.com/users/devmetal","html_url":"https://github.com/devmetal","followers_url":"https://api.github.com/users/devmetal/followers","following_url":"https://api.github.com/users/devmetal/following{/other_user}","gists_url":"https://api.github.com/users/devmetal/gists{/gist_id}","starred_url":"https://api.github.com/users/devmetal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devmetal/subscriptions","organizations_url":"https://api.github.com/users/devmetal/orgs","repos_url":"https://api.github.com/users/devmetal/repos","events_url":"https://api.github.com/users/devmetal/events{/privacy}","received_events_url":"https://api.github.com/users/devmetal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/nodejs/node/labels/memory","name":"memory","color":"eb6420"},{"url":"https://api.github.com/repos/nodejs/node/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/nodejs/node/labels/stream","name":"stream","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-06T09:06:12Z","updated_at":"2016-10-06T13:46:58Z","closed_at":null,"body":"<!--\r\nThank you for reporting an issue.\r\nPlease fill in as much of the template below as you're able.\r\n\r\nVersion: output of `node -v`\r\nPlatform: output of `uname -a` (UNIX), or version and 32 or 64-bit (Windows)\r\nSubsystem: if known, please specify affected core module name\r\n\r\nIf possible, please provide code that demonstrates the problem, keeping it as\r\nsimple and free of external dependencies as you are able.\r\n-->\r\n\r\n* **Version**: 4.6.0\r\n* **Platform**: Linux adam-System-Product-Name 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux\r\n* **Subsystem**: stream\r\n\r\n<!-- Enter your issue details below this comment. -->\r\n\r\nHi, i have a problems with stream. I try \"streamify\" some database selects. I created a pager. The pager job is simple, get some records then push in object mode to a stream. I will show the relevant parts of the debugger code, and the results. I hope this is not an issue just i miss something about streams.\r\n\r\nMy workflow is simple:\r\n[ {/*result objects from database*/}, {},  {} ] -> (Push objects to PassThrough) {}, {} -> (Create array rows) [ 'values', 'from', 'objects' ] -> (CSV Writer) -> File or http response\r\n\r\n```JavaScript\r\n//This will create arrays from specified keys from object\r\nconst recordStream = (fields) => new Transform({\r\n  readableObjectMode: true,\r\n  writableObjectMode: true,\r\n\r\n  transform(chunk, enc, done) {\r\n    if (!chunk) {\r\n      return this.end();\r\n    }\r\n\r\n    const record = [];\r\n    for (const field of fields) {\r\n      record.push(chunk[field]);\r\n    }\r\n\r\n    this.push(record);\r\n    done();\r\n  }\r\n});\r\n\r\n// this is a csv writer instance from https://www.npmjs.com/package/csv-write-stream\r\nconst csvStream = (headers) => csvWrite({ headers });\r\n\r\n\r\n// For better maintanable code, i use generators with promises\r\n\r\n/**\r\n * Based on this\r\n * https://www.promisejs.org/generators/\r\n */\r\nconst gen = function generatorAsync(iterator) {\r\n  return function generatorHandler() {\r\n    const internal = iterator.apply(this, arguments);\r\n\r\n    function handle(result) {\r\n      if (result.done) return Promise.resolve(result.value);\r\n\r\n      return Promise.resolve(result.value).then(\r\n        res => handle(internal.next(res)),\r\n        err => handle(internal.throw(err))\r\n      );\r\n    };\r\n\r\n    try {\r\n      return handle(internal.next());\r\n    } catch (e) {\r\n      return Promise.reject(e);\r\n    }\r\n  }\r\n}\r\n\r\n// The pager get stream instance and push evry record to\r\nconst pager = gen(function* (stream) {\r\n  let skip = 0;\r\n  const query = 'SELECT FROM E SKIP :skip LIMIT :limit';\r\n  let results = yield db.query(query, { params: { skip, limit: 5000 } });\r\n\r\n  while (results && !!results.length) {\r\n    for (const row of results)  {\r\n      stream.push(row);\r\n    }\r\n\r\n    skip += results.length;\r\n    results = yield db.query(query, { params: { skip, limit: 5000 } });\r\n  }\r\n\r\n  return stream.end();\r\n});\r\n\r\n\r\nconst records = recordStream(fields);\r\nconst csv = csvStream(fields);\r\nconst through = new PassThrough({ objectMode: true });\r\n\r\nthrough\r\n  .pipe(records)\r\n  .pipe(csv)\r\n  .pipe(fs.createWriteStream('./out.csv'));\r\n\r\nthrough.on('end', () => console.log('end'));\r\n\r\npager(through);\r\n\r\n// Debug\r\nsetInterval(function () {\r\n  var mem = process.memoryUsage();\r\n  var fmt = v => (v / (1024 * 1024)).toFixed(0) + 'MB';\r\n  console.log('RSS = ' + fmt(mem.rss), 'Heap = ' + fmt(mem.heapUsed));\r\n}, 1000);\r\n\r\n```\r\n\r\nThere is a lot of record in database 30.000+. But its run really fast. The problem is the leak. I runned the server via pm2 and i see in monitor the memory not released.\r\n\r\nWhen i run this code the output always similar this:\r\n\r\n```bash\r\nRSS = 80MB Heap = 29MB\r\nRSS = 92MB Heap = 50MB\r\nRSS = 102MB Heap = 55MB\r\nRSS = 108MB Heap = 60MB\r\nRSS = 101MB Heap = 28MB\r\nRSS = 101MB Heap = 41MB\r\nend\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\nRSS = 101MB Heap = 41MB\r\n...\r\n```\r\n\r\nIf you want to, i can put the whole code here. Just some requires and database connection. \r\nI have to use orientdb with orientjs. \r\n\r\nThank you for help"},"comment":{"url":"https://api.github.com/repos/nodejs/node/issues/comments/251965887","html_url":"https://github.com/nodejs/node/issues/8955#issuecomment-251965887","issue_url":"https://api.github.com/repos/nodejs/node/issues/8955","id":251965887,"user":{"login":"devmetal","id":3758485,"avatar_url":"https://avatars.githubusercontent.com/u/3758485?v=3","gravatar_id":"","url":"https://api.github.com/users/devmetal","html_url":"https://github.com/devmetal","followers_url":"https://api.github.com/users/devmetal/followers","following_url":"https://api.github.com/users/devmetal/following{/other_user}","gists_url":"https://api.github.com/users/devmetal/gists{/gist_id}","starred_url":"https://api.github.com/users/devmetal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devmetal/subscriptions","organizations_url":"https://api.github.com/users/devmetal/orgs","repos_url":"https://api.github.com/users/devmetal/repos","events_url":"https://api.github.com/users/devmetal/events{/privacy}","received_events_url":"https://api.github.com/users/devmetal/received_events","type":"User","site_admin":false},"created_at":"2016-10-06T13:46:58Z","updated_at":"2016-10-06T13:46:58Z","body":"@bnoordhuis \r\n\r\nSorry, i made a huge mistake. Actually right now i replaced the readline with my custom line be line transform stream:\r\n\r\n```JavaScript\r\nclass LineByLine extends Transform {\r\n  constructor(options) {\r\n    super(options);\r\n    this.buff = '';\r\n  }\r\n\r\n  _transform(chunk, enc, done) {\r\n    const chr = chunk.toString();\r\n    let i = 0;\r\n\r\n    while(i < chr.length) {\r\n      if (chr[i] === '\\n') {\r\n        this.push(this.buff);\r\n        this.buff = '';\r\n      } else {\r\n        this.buff += chr[i];\r\n      }\r\n\r\n      i++;\r\n    }\r\n\r\n    done();\r\n  }\r\n}\r\n```\r\n\r\nI think its not the best way to do this but its really fast. With 1000000 record its done by seconds. So sorry for this, but i think we have also a little leak. Here is the output with 1000000 record without readline:\r\n\r\n```bash\r\nRSS = 57MB Heap = 16MB\r\nRSS = 60MB Heap = 17MB\r\nRSS = 60MB Heap = 16MB\r\nRSS = 60MB Heap = 11MB\r\nRSS = 57MB Heap = 18MB\r\nRSS = 57MB Heap = 7MB\r\nRSS = 57MB Heap = 14MB\r\nRSS = 58MB Heap = 6MB\r\nRSS = 58MB Heap = 12MB\r\nRSS = 58MB Heap = 14MB\r\nRSS = 58MB Heap = 17MB\r\nRSS = 58MB Heap = 10MB\r\nclose\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\nRSS = 59MB Heap = 12MB\r\n```\r\n\r\nWhat do you think? 2MB is important or its normal?"}}
{"actor":{"display_login":"devmetal"}}