{"action":"closed","issue":{"url":"https://api.github.com/repos/archiverjs/node-archiver/issues/203","repository_url":"https://api.github.com/repos/archiverjs/node-archiver","labels_url":"https://api.github.com/repos/archiverjs/node-archiver/issues/203/labels{/name}","comments_url":"https://api.github.com/repos/archiverjs/node-archiver/issues/203/comments","events_url":"https://api.github.com/repos/archiverjs/node-archiver/issues/203/events","html_url":"https://github.com/archiverjs/node-archiver/issues/203","id":181517801,"number":203,"title":"Out of memory when zipping directory.","user":{"login":"rloehr","id":7839002,"avatar_url":"https://avatars.githubusercontent.com/u/7839002?v=3","gravatar_id":"","url":"https://api.github.com/users/rloehr","html_url":"https://github.com/rloehr","followers_url":"https://api.github.com/users/rloehr/followers","following_url":"https://api.github.com/users/rloehr/following{/other_user}","gists_url":"https://api.github.com/users/rloehr/gists{/gist_id}","starred_url":"https://api.github.com/users/rloehr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rloehr/subscriptions","organizations_url":"https://api.github.com/users/rloehr/orgs","repos_url":"https://api.github.com/users/rloehr/repos","events_url":"https://api.github.com/users/rloehr/events{/privacy}","received_events_url":"https://api.github.com/users/rloehr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-06T20:31:05Z","updated_at":"2016-12-22T02:26:14Z","closed_at":"2016-12-22T02:26:14Z","body":"Archiver has an upper bound memory limit when zipping a file.\n\nFor example,  let's say you have a million files in a directory and you attempt to zip that directory.\n`util.walkdir(dirpath, function(err, results) {\n    if (err) {\n      self.emit('error', err);\n    } else {\n      results.forEach(function(file) {\n        var entryData = _.extend({}, data);\n        entryData.name = file.relative;\n        entryData.prefix = destpath;\n        entryData.stats = file.stats;\n\n```\n    try {\n      if (dataFunction) {\n        entryData = dataFunction(entryData);\n\n        if (typeof entryData !== 'object') {\n          throw new Error('directory: invalid data returned from custom function');\n        }\n      }\n    } catch(e) {\n      self.emit('error', e);\n      return;\n    }\n\n    self._append(file.path, entryData);\n  });\n}\n\nself._pending--;\nself._maybeFinalize();\n```\n\n  });\n\n  return this;\n};`\n\nIf there happened to a plethora of files in the directory, (e.g. hundreds of thousands or a million) the results array becomes huge.\n\nThen for each result, an entry data object is created and a task is ultimately created for each entry data.   All that has to live in memory at the same time and will have a finite bound.\n\nThe task queue is a great idea but it seems like to alleviate memory pressure we should be filling it with a subset of files, burning down the queue, filling with next chunk of files....and then repeat.\n"}}
{"actor":{"display_login":"ctalkington"}}