{"action":"created","issue":{"url":"https://api.github.com/repos/expressjs/multer/issues/357","repository_url":"https://api.github.com/repos/expressjs/multer","labels_url":"https://api.github.com/repos/expressjs/multer/issues/357/labels{/name}","comments_url":"https://api.github.com/repos/expressjs/multer/issues/357/comments","events_url":"https://api.github.com/repos/expressjs/multer/issues/357/events","html_url":"https://github.com/expressjs/multer/issues/357","id":162493852,"number":357,"title":"Add core stream storage driver","user":{"login":"wesleytodd","id":1027776,"avatar_url":"https://avatars.githubusercontent.com/u/1027776?v=3","gravatar_id":"","url":"https://api.github.com/users/wesleytodd","html_url":"https://github.com/wesleytodd","followers_url":"https://api.github.com/users/wesleytodd/followers","following_url":"https://api.github.com/users/wesleytodd/following{/other_user}","gists_url":"https://api.github.com/users/wesleytodd/gists{/gist_id}","starred_url":"https://api.github.com/users/wesleytodd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesleytodd/subscriptions","organizations_url":"https://api.github.com/users/wesleytodd/orgs","repos_url":"https://api.github.com/users/wesleytodd/repos","events_url":"https://api.github.com/users/wesleytodd/events{/privacy}","received_events_url":"https://api.github.com/users/wesleytodd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-06-27T16:30:25Z","updated_at":"2016-06-28T19:12:42Z","closed_at":null,"body":"So I am working with the \"transform file\" branch from #356 and I realized that what I really want is just a stream storage driver.  Basically it would be one that just doesn't read from the stream at all and relies on you doing that in your handler method.  What do you think about this?\r\n\r\nHere is basically what I was thinking it would look like:\r\n\r\n```javascript\r\nfunction StreamStorage (opts) {}\r\n\r\nStreamStorage.prototype._handleFile = function _handleFile (req, file, cb) {\r\n  cb(null, {})\r\n}\r\n\r\nStreamStorage.prototype._removeFile = function _removeFile (req, file, cb) {\r\n  // Not sure if we would need to flush out the stream or not here...\r\n  cb(null)\r\n}\r\n\r\nmodule.exports = function (opts) {\r\n  return new StreamStorage(opts)\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/expressjs/multer/issues/comments/229152023","html_url":"https://github.com/expressjs/multer/issues/357#issuecomment-229152023","issue_url":"https://api.github.com/repos/expressjs/multer/issues/357","id":229152023,"user":{"login":"LinusU","id":189580,"avatar_url":"https://avatars.githubusercontent.com/u/189580?v=3","gravatar_id":"","url":"https://api.github.com/users/LinusU","html_url":"https://github.com/LinusU","followers_url":"https://api.github.com/users/LinusU/followers","following_url":"https://api.github.com/users/LinusU/following{/other_user}","gists_url":"https://api.github.com/users/LinusU/gists{/gist_id}","starred_url":"https://api.github.com/users/LinusU/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LinusU/subscriptions","organizations_url":"https://api.github.com/users/LinusU/orgs","repos_url":"https://api.github.com/users/LinusU/repos","events_url":"https://api.github.com/users/LinusU/events{/privacy}","received_events_url":"https://api.github.com/users/LinusU/received_events","type":"User","site_admin":false},"created_at":"2016-06-28T19:12:42Z","updated_at":"2016-06-28T19:12:42Z","body":"Okay so during the last couple of months I've been thinking a lot of the api we expose, how it works with the storage modules and how easy it is to consume.\r\n\r\nThe first sign that things weren't the best they could was when #303 was opened. This was implemented as a storage module even though it just wanted to modify the stream.\r\n\r\nWe also have a lot of callbacks option that feels a bit too much, e.g. should we really have a callback for the directory, and one for the filename? For some apps that makes sense though, but for some it doesn't.\r\n\r\nI have also been looking a bit at how other libs/frameworks does it, and how things are being used generally in the Node.js community.\r\n\r\nWhat I've started to feel is that streams are a very good way of thinking about the \"files\" that are being uploaded, but that we currently do too much job in making sure that it's as effective as possible. This makes us loose in usability, and also hurts some scenarios. E.g. with the S3 module you can be forced to upload something for it just to be removed because of some kind of error, this costs money. In cases like that I would have preferred it to use some memory or temporary disk space.\r\n\r\nAnother big use case is doing some kind of validation on the file and then sending it to, e.g. S3. When you decide to add the validations, you can no longer use the multer-s3 module. This, in my opinion, takes a way a lot of the power of multer. It might be easy to get started with, but it doesn't always scale with the user.\r\n\r\nWhat I would like to discuss moving towards, would be an api where we wouldn't give the users the ability to choose a storage engine. Instead we would handle \"accepting\" the files and storing them temporarily. That could be done in memory or, when the file size hits a threshold, on disk.\r\n\r\nWe could then expose information about the file together with a stream, that could be consumed however the end user see fit. It could be transformed, stored on disk, validated or uploaded to a third party service.\r\n\r\nI imagine the interface looking something like this.\r\n\r\n```ts\r\ninterface File {\r\n  fieldname: string\r\n  filename: string\r\n  encoding: string\r\n  mimetype: string\r\n  size: number\r\n  stream: ReadableStream\r\n}\r\n```\r\n\r\nThe user would then be given infinite opportunities to do whatever they want to the file. e.g.\r\n\r\nSave it to disk\r\n```js\r\nreq.file.stream\r\n  .pipe(fs.createWriteStream('uploads/test.png'))\r\n```\r\n\r\nEncrypt it and upload to S3\r\n```js\r\nreq.file.stream\r\n  .pipe(crypto.createCipher('aes-256-gcm', 'test-key'))\r\n  .pipe(uploadFileToS3('test.png'))\r\n```\r\n\r\nLoad image, resize it and save to disk\r\n```js\r\nreq.file.stream.pipe(concat(buffer => {\r\n  sharp(buffer)\r\n    .resize(128, 128)\r\n    .toFile('uploads/test.png')\r\n}))\r\n```\r\n\r\nWe could potentially provide some utility functions to make saving the raw file to disk easier, and we could also speed it up by only moving the file if it's already on disk. This would be similar to the famous php function [`move_uploaded_file`](http://php.net/manual/en/function.move-uploaded-file.php).\r\n\r\n```js\r\nmulter.saveFileAs(req.file, 'uploads/test.png', err => {\r\n  if (err) return next(err)\r\n\r\n  res.send('Image uploaded!')\r\n})\r\n```\r\n\r\nAny thoughts on this would be highly appreciated.\r\n\r\nping @hacksparrow @jpfluger @Fishrock123 @gabeio (I know I missed someone, sorry)"}}
{"actor":{"display_login":"LinusU"}}