{"action":"opened","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/17179","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17179/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17179/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17179/events","html_url":"https://github.com/elastic/elasticsearch/issues/17179","id":141768944,"number":17179,"title":"Cloud-AWS: S3 Repository Across-Account fails verification.","user":{"login":"JamesBromberger","id":1736320,"avatar_url":"https://avatars.githubusercontent.com/u/1736320?v=3","gravatar_id":"","url":"https://api.github.com/users/JamesBromberger","html_url":"https://github.com/JamesBromberger","followers_url":"https://api.github.com/users/JamesBromberger/followers","following_url":"https://api.github.com/users/JamesBromberger/following{/other_user}","gists_url":"https://api.github.com/users/JamesBromberger/gists{/gist_id}","starred_url":"https://api.github.com/users/JamesBromberger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JamesBromberger/subscriptions","organizations_url":"https://api.github.com/users/JamesBromberger/orgs","repos_url":"https://api.github.com/users/JamesBromberger/repos","events_url":"https://api.github.com/users/JamesBromberger/events{/privacy}","received_events_url":"https://api.github.com/users/JamesBromberger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-18T03:55:41Z","updated_at":"2016-03-18T03:55:41Z","closed_at":null,"body":"**Elasticsearch version**: 2.2.0\r\n\r\n**JVM version**: 1.8.0\r\n\r\n**OS version**: Amazon Linux 2015.09.02\r\n\r\n**Description of the problem including expected versus actual behavior**: Using Cloud-AWS Plugin to define a repository that is across AWS accounts for escrow backps/snapshots\r\n\r\n**Steps to reproduce**:\r\n 1. Create two AWS accounts, A for our primary workload (ElasticSearch), and B for our escrow copy (think: security team)\r\n 2. Create a bucket in B (security-bucket), with an S3 Bucket Policy that permits account A to write to a prefix in this bucket, with a condition that the submitter (in A) set the ACL \"bucket-owner-full-control'. Also permit account A to list and get objects from this prefix, but not DELETE. (NB: does permit overwrite, but S3Versioning can help here - another topic)\r\n 3. In account A, create an IAM Role for EC2 instance that permits access to B's security-bucket, as well as the normal EC2 discovery requirements.\r\n 4. Launch an EC2 instance running ElasticSearch in this Role.\r\n 5. Configure the ElasticSearch instance to have a repository of B's security-bucket\r\n\r\n**Provide logs (if relevant)**:\r\n`{\"error\":{\"root_cause\":[{\"type\":\"repository_verification_exception\",\"reason\":\"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node\"}],\"type\":\"repository_verification_exception\",\"reason\":\"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node\",\"caused_by\":{\"type\":\"i_o_exception\",\"reason\":\"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp\",\"caused_by\":{\"type\":\"amazon_s3_exception\",\"reason\":\"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)\"}}},\"status\":500}`\r\n\r\n**Describe the feature**:\r\nI am using ES 2.2.0 , and setting up a cross account trust where the receiving bucket (in another AWS account0 requires \"bucket-owner-full-control\", Using AWS CLi confirms I can put objects with the canned ACl into the prefix I have reserved for this; I can list the objects back, and I can read the content back.\r\n\r\nHowever, when trying to set up a backup location in elasticsearch, it fails the verification. My config is:\r\n`{ \"type\": \"s3\", \"settings\": { \"bucket\": \"XXX-logs\", \"region\": \"ap-southeast-2\", \"server_side_encryption\": \"True\", \"base_path\": \"XXX/Dev/Elasticsearch/\", \"buffer_size\": \"100mb\", \"canned_acl\": \"bucket-owner-full-control\" } }`\r\n\r\nHowever when submitting this I get:\r\n`{\"error\":{\"root_cause\":[{\"type\":\"repository_verification_exception\",\"reason\":\"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node\"}],\"type\":\"repository_verification_exception\",\"reason\":\"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node\",\"caused_by\":{\"type\":\"i_o_exception\",\"reason\":\"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp\",\"caused_by\":{\"type\":\"amazon_s3_exception\",\"reason\":\"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)\"}}},\"status\":500}`\r\n\r\nWhich from the S3 logs on the target S3 bucket I see a 403 AccessDenied. The only condition that is required by S3 in my case is the bucket-owner-full-control ACL. It appears that the check in index/snapshots/blobstore/BlobStoreIndexShardRepository.java calling org.elasticsearch.repositories.blobstore.BlobStoreRepository.testBlobPrefix() may not have the canned ACL being added, and thus fails to get set up.\r\n\r\nThis maybe somewhere around startVerification() in repositories/blobstore/BlobStoreRepository.java.\r\n\r\nIn my use case, the target bucket is to be write once (no deletes; escrow copy of the backup, so the corresponding \"ensure moves are supported\" is also likely to fail....\r\n"}}