{"action":"created","issue":{"url":"https://api.github.com/repos/ncw/rclone/issues/415","repository_url":"https://api.github.com/repos/ncw/rclone","labels_url":"https://api.github.com/repos/ncw/rclone/issues/415/labels{/name}","comments_url":"https://api.github.com/repos/ncw/rclone/issues/415/comments","events_url":"https://api.github.com/repos/ncw/rclone/issues/415/events","html_url":"https://github.com/ncw/rclone/issues/415","id":145186539,"number":415,"title":"S3 to S3 copy ends up with TotalPartsExceeded","user":{"login":"dkorunic","id":3041939,"avatar_url":"https://avatars.githubusercontent.com/u/3041939?v=3","gravatar_id":"","url":"https://api.github.com/users/dkorunic","html_url":"https://github.com/dkorunic","followers_url":"https://api.github.com/users/dkorunic/followers","following_url":"https://api.github.com/users/dkorunic/following{/other_user}","gists_url":"https://api.github.com/users/dkorunic/gists{/gist_id}","starred_url":"https://api.github.com/users/dkorunic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dkorunic/subscriptions","organizations_url":"https://api.github.com/users/dkorunic/orgs","repos_url":"https://api.github.com/users/dkorunic/repos","events_url":"https://api.github.com/users/dkorunic/events{/privacy}","received_events_url":"https://api.github.com/users/dkorunic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-04-01T13:53:06Z","updated_at":"2016-04-05T18:30:44Z","closed_at":null,"body":"Hi,\r\n\r\nWe're doing a S3 to S3 copy with rclone and we have 2.6 GB files ending up with TotalPartsExceeded error:\r\n\r\n2016/04/01 15:48:02 D156748_1.mp4: Failed to copy: MultipartUpload: upload multipart failed\r\n        upload id: 2~dxDK4rkYSz7eIt40U_wqcoz6xh4oCZL\r\ncaused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit\r\n\r\nBoth S3 storage backends are Ceph/RadosGW, if that helps. Rclone being used is v1.28 (tried -v1.28-8-ga1323ebβ and v1.28-9-g9dccf91β, neither seem capable of finishing larger uploads).\r\n\r\nCommand being used is simple rclone copy s3_old:bucket s3_new:bucket. Let me know if you need full rclone -v output.\r\n"},"comment":{"url":"https://api.github.com/repos/ncw/rclone/issues/comments/205934662","html_url":"https://github.com/ncw/rclone/issues/415#issuecomment-205934662","issue_url":"https://api.github.com/repos/ncw/rclone/issues/415","id":205934662,"user":{"login":"dkorunic","id":3041939,"avatar_url":"https://avatars.githubusercontent.com/u/3041939?v=3","gravatar_id":"","url":"https://api.github.com/users/dkorunic","html_url":"https://github.com/dkorunic","followers_url":"https://api.github.com/users/dkorunic/followers","following_url":"https://api.github.com/users/dkorunic/following{/other_user}","gists_url":"https://api.github.com/users/dkorunic/gists{/gist_id}","starred_url":"https://api.github.com/users/dkorunic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dkorunic/subscriptions","organizations_url":"https://api.github.com/users/dkorunic/orgs","repos_url":"https://api.github.com/users/dkorunic/repos","events_url":"https://api.github.com/users/dkorunic/events{/privacy}","received_events_url":"https://api.github.com/users/dkorunic/received_events","type":"User","site_admin":false},"created_at":"2016-04-05T18:30:44Z","updated_at":"2016-04-05T18:30:44Z","body":"@ncw Huh. Even with -v and logging everything to file, there aren't any such strings in the output. What I get is at best:\r\n\r\n *                                   D211028.mxf: 99% done. avg: 1379.9, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1366.0, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1352.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1338.8, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1325.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1312.7, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1300.1, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1287.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1275.5, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1263.5, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1251.8, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1240.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1228.9, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1217.8, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1206.9, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1196.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1185.7, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1175.3, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1165.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1155.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D211028.mxf: 99% done. avg: 1145.4, cur:    0.0 kByte/s. ETA: 0s\r\n...\r\n2016/04/05 19:57:57 D211028.mxf: Failed to copy: MultipartUpload: upload multipart failed\r\n        upload id: 2~IQkBzTaPq7AVS50Dajb8UF-KiqD6asm\r\ncaused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit\r\n2016/04/05 19:57:57 D211028.mxf: Removing failed copy\r\n\r\nor similar:\r\n\r\n *                                   D176573.mxf: 98% done. avg:  208.9, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  205.7, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  202.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  199.5, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  196.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  193.7, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  191.0, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  188.3, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  185.7, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  183.1, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  180.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  178.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  175.9, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  173.6, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  171.4, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  169.2, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  167.1, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  165.0, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  163.0, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  161.0, cur:    0.0 kByte/s. ETA: 0s\r\n *                                   D176573.mxf: 98% done. avg:  159.1, cur:    0.0 kByte/s. ETA: 0s\r\n...\r\n2016/04/05 20:25:17 D176573.mxf: Failed to copy: MultipartUpload: upload multipart failed\r\n        upload id: 2~IQJmqAGSijxX6ldZWRRK68dShmnKM80\r\ncaused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit\r\n2016/04/05 20:25:17 D176573.mxf: Removing failed copy\r\n\r\nThe set of parameters and arguments I'm using rclone with is fairly trivial:\r\n\r\n./rclone -v --checkers 40 --no-gzip-encoding --retries 1 --transfers 60 --quiet sync store1_s3:Video store2_s3:Video\r\n\r\nAny ideas?"}}