{"action":"created","issue":{"url":"https://api.github.com/repos/minio/minio/issues/2334","repository_url":"https://api.github.com/repos/minio/minio","labels_url":"https://api.github.com/repos/minio/minio/issues/2334/labels{/name}","comments_url":"https://api.github.com/repos/minio/minio/issues/2334/comments","events_url":"https://api.github.com/repos/minio/minio/issues/2334/events","html_url":"https://github.com/minio/minio/issues/2334","id":168470966,"number":2334,"title":"Don't concatenate multi-part upload during completion","user":{"login":"ramondeklein","id":6048186,"avatar_url":"https://avatars.githubusercontent.com/u/6048186?v=3","gravatar_id":"","url":"https://api.github.com/users/ramondeklein","html_url":"https://github.com/ramondeklein","followers_url":"https://api.github.com/users/ramondeklein/followers","following_url":"https://api.github.com/users/ramondeklein/following{/other_user}","gists_url":"https://api.github.com/users/ramondeklein/gists{/gist_id}","starred_url":"https://api.github.com/users/ramondeklein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ramondeklein/subscriptions","organizations_url":"https://api.github.com/users/ramondeklein/orgs","repos_url":"https://api.github.com/users/ramondeklein/repos","events_url":"https://api.github.com/users/ramondeklein/events{/privacy}","received_events_url":"https://api.github.com/users/ramondeklein/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/minio/minio/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/minio/minio/labels/urgent","name":"urgent","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"harshavardhana","id":622699,"avatar_url":"https://avatars.githubusercontent.com/u/622699?v=3","gravatar_id":"","url":"https://api.github.com/users/harshavardhana","html_url":"https://github.com/harshavardhana","followers_url":"https://api.github.com/users/harshavardhana/followers","following_url":"https://api.github.com/users/harshavardhana/following{/other_user}","gists_url":"https://api.github.com/users/harshavardhana/gists{/gist_id}","starred_url":"https://api.github.com/users/harshavardhana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshavardhana/subscriptions","organizations_url":"https://api.github.com/users/harshavardhana/orgs","repos_url":"https://api.github.com/users/harshavardhana/repos","events_url":"https://api.github.com/users/harshavardhana/events{/privacy}","received_events_url":"https://api.github.com/users/harshavardhana/received_events","type":"User","site_admin":false},"assignees":[{"login":"harshavardhana","id":622699,"avatar_url":"https://avatars.githubusercontent.com/u/622699?v=3","gravatar_id":"","url":"https://api.github.com/users/harshavardhana","html_url":"https://github.com/harshavardhana","followers_url":"https://api.github.com/users/harshavardhana/followers","following_url":"https://api.github.com/users/harshavardhana/following{/other_user}","gists_url":"https://api.github.com/users/harshavardhana/gists{/gist_id}","starred_url":"https://api.github.com/users/harshavardhana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshavardhana/subscriptions","organizations_url":"https://api.github.com/users/harshavardhana/orgs","repos_url":"https://api.github.com/users/harshavardhana/repos","events_url":"https://api.github.com/users/harshavardhana/events{/privacy}","received_events_url":"https://api.github.com/users/harshavardhana/received_events","type":"User","site_admin":false}],"milestone":null,"comments":11,"created_at":"2016-07-30T14:52:10Z","updated_at":"2016-08-31T20:54:24Z","closed_at":null,"body":"The FS version concatenates all the parts of a multi-part upload when the multi-upload is completed. Multi-part uploads are often used for huge files, so this takes a lot of time and also spikes the IOPS for a while. In some situations (like ours), it would be much better to use the XL method to keep the parts separate and combine them during download.\r\n\r\nOf course, the parts cannot be accessed the regular way (via the filesystem) anymore. But most systems that use S3 only use S3 to access them (you can't do otherwise in AWS), so this wouldn't be a big problem.\r\n\r\nConcatenating files might be the sensible default option, but when performance and I/O is critical, then keeping the files separate might be the better option. Now, we reverted using the XL method, solely for this feature, but this costs us a lot of additional disk space."},"comment":{"url":"https://api.github.com/repos/minio/minio/issues/comments/243898196","html_url":"https://github.com/minio/minio/issues/2334#issuecomment-243898196","issue_url":"https://api.github.com/repos/minio/minio/issues/2334","id":243898196,"user":{"login":"harshavardhana","id":622699,"avatar_url":"https://avatars.githubusercontent.com/u/622699?v=3","gravatar_id":"","url":"https://api.github.com/users/harshavardhana","html_url":"https://github.com/harshavardhana","followers_url":"https://api.github.com/users/harshavardhana/followers","following_url":"https://api.github.com/users/harshavardhana/following{/other_user}","gists_url":"https://api.github.com/users/harshavardhana/gists{/gist_id}","starred_url":"https://api.github.com/users/harshavardhana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshavardhana/subscriptions","organizations_url":"https://api.github.com/users/harshavardhana/orgs","repos_url":"https://api.github.com/users/harshavardhana/repos","events_url":"https://api.github.com/users/harshavardhana/events{/privacy}","received_events_url":"https://api.github.com/users/harshavardhana/received_events","type":"User","site_admin":false},"created_at":"2016-08-31T20:54:24Z","updated_at":"2016-08-31T20:54:24Z","body":"@ramondeklein we have merged a change in master which preempts the concatenation for FS backend. \r\nWhich would in-turn avoid the larger delay at the end of a complete multipart upload.  \r\n\r\nWould you validate 'github.com/minio/minio/' 'master' ? "}}
{"actor":{"display_login":"harshavardhana"}}