{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-cli/issues/1752","repository_url":"https://api.github.com/repos/aws/aws-cli","labels_url":"https://api.github.com/repos/aws/aws-cli/issues/1752/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-cli/issues/1752/comments","events_url":"https://api.github.com/repos/aws/aws-cli/issues/1752/events","html_url":"https://github.com/aws/aws-cli/issues/1752","id":129371385,"number":1752,"title":"Bug while sync'ing from S3 to local server using awscli","user":{"login":"alanjohnmiller","id":11881330,"avatar_url":"https://avatars.githubusercontent.com/u/11881330?v=3","gravatar_id":"","url":"https://api.github.com/users/alanjohnmiller","html_url":"https://github.com/alanjohnmiller","followers_url":"https://api.github.com/users/alanjohnmiller/followers","following_url":"https://api.github.com/users/alanjohnmiller/following{/other_user}","gists_url":"https://api.github.com/users/alanjohnmiller/gists{/gist_id}","starred_url":"https://api.github.com/users/alanjohnmiller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanjohnmiller/subscriptions","organizations_url":"https://api.github.com/users/alanjohnmiller/orgs","repos_url":"https://api.github.com/users/alanjohnmiller/repos","events_url":"https://api.github.com/users/alanjohnmiller/events{/privacy}","received_events_url":"https://api.github.com/users/alanjohnmiller/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/aws/aws-cli/labels/closing-soon-if-no-response","name":"closing-soon-if-no-response","color":"5319e7"},{"url":"https://api.github.com/repos/aws/aws-cli/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/aws/aws-cli/labels/s3","name":"s3","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"kyleknap","id":4605355,"avatar_url":"https://avatars.githubusercontent.com/u/4605355?v=3","gravatar_id":"","url":"https://api.github.com/users/kyleknap","html_url":"https://github.com/kyleknap","followers_url":"https://api.github.com/users/kyleknap/followers","following_url":"https://api.github.com/users/kyleknap/following{/other_user}","gists_url":"https://api.github.com/users/kyleknap/gists{/gist_id}","starred_url":"https://api.github.com/users/kyleknap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyleknap/subscriptions","organizations_url":"https://api.github.com/users/kyleknap/orgs","repos_url":"https://api.github.com/users/kyleknap/repos","events_url":"https://api.github.com/users/kyleknap/events{/privacy}","received_events_url":"https://api.github.com/users/kyleknap/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2016-01-28T07:50:45Z","updated_at":"2016-02-03T23:49:55Z","closed_at":"2016-02-03T23:49:55Z","body":"I have an s3 bucket with hundreds of folders, and thousands of files. It is also a static website.\r\n\r\nI sync the bucket to a local CentOS 6.7 server which is also a running the same website. \r\nThe server is installed with:\r\naws-cli/1.10.0 Python/2.7.9 Linux/2.6.32-573.8.1.el6.x86_64 botocore/1.3.22\r\n\r\nI run the following command:\r\n\r\naws s3 sync s3://www.website.com/ /home/website/public_html/ --delete --exact-timestamps\r\n\r\nWhich seems to correctly sync the entire website, but outputs the following errors:\r\n\r\ndownload failed: s3://www.website.com/www/ to ../../../home/website/public_html/www/ [Errno 21] Is a directory: u'/home/website/public_html/www/'\r\ndownload failed: s3://www.website.com/www/en/html/ to ../../../home/website/public_html/www/en/html/ [Errno 21] Is a directory: u'/home/website/public_html/www/en/html/'\r\nCompleted 2 part(s) with ... file(s) remaining\r\n\r\nIt does not report the same problem on any other local folder, even though it is syncing hundreds of folders.\r\n\r\nI am trying to work out why this happens, and what the error means. \r\n\r\n/home/website/public_html/www/\r\n/home/website/public_html/www/en/html/\r\nare local folders. So the error message makes no sense at all.\r\n\r\nThe local permissions of these folders are the same as all other folders that are processed correctly without errors.\r\nThe S3 permissions of these folders are the same as all other folders processed correctly from S3 without errors.\r\n\r\nThis appears to be a bug, and I do not know why it reports an error when all other folders are correctly processed.\r\n\r\nAny help would be appreciated."},"comment":{"url":"https://api.github.com/repos/aws/aws-cli/issues/comments/179536203","html_url":"https://github.com/aws/aws-cli/issues/1752#issuecomment-179536203","issue_url":"https://api.github.com/repos/aws/aws-cli/issues/1752","id":179536203,"user":{"login":"alanjohnmiller","id":11881330,"avatar_url":"https://avatars.githubusercontent.com/u/11881330?v=3","gravatar_id":"","url":"https://api.github.com/users/alanjohnmiller","html_url":"https://github.com/alanjohnmiller","followers_url":"https://api.github.com/users/alanjohnmiller/followers","following_url":"https://api.github.com/users/alanjohnmiller/following{/other_user}","gists_url":"https://api.github.com/users/alanjohnmiller/gists{/gist_id}","starred_url":"https://api.github.com/users/alanjohnmiller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanjohnmiller/subscriptions","organizations_url":"https://api.github.com/users/alanjohnmiller/orgs","repos_url":"https://api.github.com/users/alanjohnmiller/repos","events_url":"https://api.github.com/users/alanjohnmiller/events{/privacy}","received_events_url":"https://api.github.com/users/alanjohnmiller/received_events","type":"User","site_admin":false},"created_at":"2016-02-03T23:49:55Z","updated_at":"2016-02-03T23:49:55Z","body":"I have resolved my original issue.\r\n\r\nEven though Amazon itself, and S3 Browser (a windows product for checking permissions on S3 objects) reported 0 bytes, and / on the end of each sub-folder, I found that for the two problem folders, there actually was 1 byte stored in each S3 object.\r\n\r\nI used another product for windows (S3 Express) to upload folders only if they have changed, and the two folders with errors were uploaded correctly and corrected to 0 byte entries.\r\n\r\nIf anyone is using S3 Express in Windows, the command that fixed the folders was:\r\n\r\nput D:\\local\\top\\of\\tree\\folder\\ s3bucket/top/of/tree/folder/ -include:\"*.htm|*.html\" -exclude:\"*.*\" -meta:\"cache-control:max-age=43200\" -s -onlynewer -purge -showdelete -showfiles -t:16 -optimize\r\n\r\nBasically this command re-uploads my html, while excluding uploading everything else (which means I did not have to upload 12GB of data again), getting rid of any folder on S3 that does not match. If you use the -sim option in S3 Express, it will simulate so that you can see what will happen.\r\n\r\nIt appears that sometimes during an upload to S3, a single byte entry is made to some folders (I don't know why this happens yet), which turns the object on S3 into a file, which then causes the error described in the beginning of this issue while downloading to another location. However, these 1 byte entries are not correctly reported in Amazon itself or most other tools. Once these single byte objects are corrected, awscli then also operates correctly.\r\n\r\nHowever, I could not fix the problem using awscli. I had to fix the problem with S3 Express. Perhaps in a future enhancement to awscli, there could be some kind of command which finds S3 objects that have / after them, and are 1 byte, and reports them as potential folder errors, and/or fixes them?\r\n\r\n"}}