{"action":"closed","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/34074","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/34074/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/34074/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/34074/events","html_url":"https://github.com/saltstack/salt/issues/34074","id":160773396,"number":34074,"title":"Unable to use S3 file backend with 2016.3.1 on Ubuntu 14.04 or 16.04","user":{"login":"fooka03","id":1747807,"avatar_url":"https://avatars.githubusercontent.com/u/1747807?v=3","gravatar_id":"","url":"https://api.github.com/users/fooka03","html_url":"https://github.com/fooka03","followers_url":"https://api.github.com/users/fooka03/followers","following_url":"https://api.github.com/users/fooka03/following{/other_user}","gists_url":"https://api.github.com/users/fooka03/gists{/gist_id}","starred_url":"https://api.github.com/users/fooka03/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fooka03/subscriptions","organizations_url":"https://api.github.com/users/fooka03/orgs","repos_url":"https://api.github.com/users/fooka03/repos","events_url":"https://api.github.com/users/fooka03/events{/privacy}","received_events_url":"https://api.github.com/users/fooka03/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Critical","name":"Critical","color":"e95800"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Fixed%20Pending%20Verification","name":"Fixed Pending Verification","color":"d4c5f9"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P2","name":"P2","color":"155fb5"},{"url":"https://api.github.com/repos/saltstack/salt/labels/RIoT","name":"RIoT","color":"fef2c0"},{"url":"https://api.github.com/repos/saltstack/salt/labels/TEAM%20RIoT","name":"TEAM RIoT","color":"A9A9A9"}],"state":"closed","locked":false,"assignee":{"login":"gtmanfred","id":732321,"avatar_url":"https://avatars.githubusercontent.com/u/732321?v=3","gravatar_id":"","url":"https://api.github.com/users/gtmanfred","html_url":"https://github.com/gtmanfred","followers_url":"https://api.github.com/users/gtmanfred/followers","following_url":"https://api.github.com/users/gtmanfred/following{/other_user}","gists_url":"https://api.github.com/users/gtmanfred/gists{/gist_id}","starred_url":"https://api.github.com/users/gtmanfred/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gtmanfred/subscriptions","organizations_url":"https://api.github.com/users/gtmanfred/orgs","repos_url":"https://api.github.com/users/gtmanfred/repos","events_url":"https://api.github.com/users/gtmanfred/events{/privacy}","received_events_url":"https://api.github.com/users/gtmanfred/received_events","type":"User","site_admin":false},"assignees":[{"login":"gtmanfred","id":732321,"avatar_url":"https://avatars.githubusercontent.com/u/732321?v=3","gravatar_id":"","url":"https://api.github.com/users/gtmanfred","html_url":"https://github.com/gtmanfred","followers_url":"https://api.github.com/users/gtmanfred/followers","following_url":"https://api.github.com/users/gtmanfred/following{/other_user}","gists_url":"https://api.github.com/users/gtmanfred/gists{/gist_id}","starred_url":"https://api.github.com/users/gtmanfred/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gtmanfred/subscriptions","organizations_url":"https://api.github.com/users/gtmanfred/orgs","repos_url":"https://api.github.com/users/gtmanfred/repos","events_url":"https://api.github.com/users/gtmanfred/events{/privacy}","received_events_url":"https://api.github.com/users/gtmanfred/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/84","html_url":"https://github.com/saltstack/salt/milestones/C%207","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/84/labels","id":1811671,"number":84,"title":"C 7","description":"","creator":{"login":"meggiebot","id":12242451,"avatar_url":"https://avatars.githubusercontent.com/u/12242451?v=3","gravatar_id":"","url":"https://api.github.com/users/meggiebot","html_url":"https://github.com/meggiebot","followers_url":"https://api.github.com/users/meggiebot/followers","following_url":"https://api.github.com/users/meggiebot/following{/other_user}","gists_url":"https://api.github.com/users/meggiebot/gists{/gist_id}","starred_url":"https://api.github.com/users/meggiebot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meggiebot/subscriptions","organizations_url":"https://api.github.com/users/meggiebot/orgs","repos_url":"https://api.github.com/users/meggiebot/repos","events_url":"https://api.github.com/users/meggiebot/events{/privacy}","received_events_url":"https://api.github.com/users/meggiebot/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":1,"state":"open","created_at":"2016-06-06T18:57:12Z","updated_at":"2016-06-29T20:47:34Z","due_on":"2016-07-15T06:00:00Z","closed_at":null},"comments":16,"created_at":"2016-06-16T22:14:56Z","updated_at":"2016-06-29T20:47:34Z","closed_at":"2016-06-29T20:47:34Z","body":"### Description of Issue/Question\r\nAfter upgrading to 2016.3.1, running states with source files located in a s3 bucket configured as a fileserver backend would fail with the message: `Source file salt://myfile not found`  Explicitly syncing the fileserver cache with `salt-run fileserver.update backend=s3fs` results in no files being downloaded and a warning in the logs: `[salt.loaded.int.fileserver.s3fs][WARNING ][26138] S3 Error! Do you have any files in your S3 bucket?`  Downgrading to 2016.3.0 resolved the issue.  This issue is present on both Ubuntu 14.04 and 16.04.  Using AWS credentials or an IAM role both exhibit the issue.  It appears to be caused by PR #33682 (looks like the only related changes in this release)\r\n\r\n### Setup\r\n##### Master Config\r\n```\r\nmaster_id: <redacted>\r\nmax_open_files: 65000\r\nworker_threads: 16\r\ntimeout: 30\r\n\r\nfile_roots:\r\n  base:\r\n    - /srv/salt/base\r\n  development:\r\n    - /srv/salt/dev\r\n  prod:\r\n    - /srv/salt/prod\r\n\r\ntop_file_merging_strategy: same\r\ndefault_top: base\r\nhash_type: sha512\r\nfile_ignore_regex:\r\n  - '/\\.svn($|/)'\r\n  - '/\\.git($|/)'\r\n  - '/\\.hg($|/)'\r\n  - '/\\.npm($|/)'\r\n\r\ngitfs_provider: gitpython\r\n\r\nfileserver_backend:\r\n  - roots\r\n  - git\r\n  - s3fs\r\n\r\ns3.buckets:\r\n  - <redacted>\r\n\r\npillar_roots:\r\n  base:\r\n    - /srv/pillar/base\r\n  development:\r\n    - /srv/pillar/dev\r\n  prod:\r\n    - /srv/pillar/prod\r\n\r\nrest_tornado:\r\n    port: <redacted>\r\n    address: 0.0.0.0\r\n    backlog: 128\r\n    debug: False\r\n    disable_ssl: True\r\n    webhook_disable_auth: False\r\n```\r\n##### Minion Config:\r\n```\r\nmaster: 127.0.0.1\r\nrandom_master: True\r\nhash_type: sha512\r\n\r\ngrains:\r\n  role: saltmaster\r\n  deployment: <redacted>\r\n  profiles:\r\n    - salt\r\n\r\nmaster_finger: <redacted>\r\n```\r\n##### Machine Type:\r\nThe machine is a m4.large (2 cores, 8GB RAM) running in AWS East with an IAM role allowing for full S3 access.  It is only running the salt-master and a salt-minion for itself.\r\n\r\n##### Sample State File:\r\n```\r\n/etc/collectd/collectd.conf:\r\n  file.managed:\r\n    - source: salt://collectd/collectd.tmpl   #This is located in s3://<redacted>/base/collectd/collectd.tmpl\r\n    - template: jinja\r\n```\r\n##### S3 Bucket Layout:\r\n```\r\n$ aws s3 ls s3://<redacted>\r\n                           PRE base/\r\n                           PRE development/\r\n                           PRE prod/\r\n```\r\n##### Result:\r\n```\r\n----------\r\n          ID: /etc/collectd/collectd.conf\r\n    Function: file.managed\r\n      Result: False\r\n     Comment: Source file salt://collectd/collectd.tmpl not found\r\n     Started: 19:23:04.849051\r\n    Duration: 736.108 ms\r\n     Changes:   \r\n----------\r\n```\r\n##### Expected (Actual output after downgrading to 2016.3.0):\r\n```\r\n----------\r\n          ID: /etc/collectd/collectd.conf\r\n    Function: file.managed\r\n      Result: True\r\n     Comment: File /etc/collectd/collectd.conf updated\r\n     Started: 21:48:34.508841\r\n    Duration: 2277.07 ms\r\n     Changes:   \r\n              ----------\r\n              diff: <redacted>\r\n----------\r\n```\r\n### Steps to Reproduce Issue\r\nThis happened with probably the most basic use case for an s3 file backend, and on a brand new AWS EC2 instance (using AMI: ami-13be557e).  Upload file to s3://my-bucket/base/, configure master to use s3fs file backend, apply a state file that uses the uploaded file as source.\r\n\r\nI dug into the code a bit and added some logging to figure out what was coming back from S3.  First I added logging to output the s3_meta keys and the resulting meta_response object in [fileserver/s3fs.py](https://github.com/saltstack/salt/blob/develop/salt/fileserver/s3fs.py#L468) and ended up with this:\r\n```\r\n[salt.utils.lazy  ][DEBUG   ][14199] LazyLoaded s3fs.envs\r\n[salt.fileserver  ][DEBUG   ][14199] Updating s3fs fileserver cache\r\n[salt.loaded.int.fileserver.s3fs][DEBUG   ][14199] Refreshing buckets cache file\r\n[requests.packages.urllib3.connectionpool][INFO    ][14199] Starting new HTTP connection (1): 169.254.169.254\r\n[requests.packages.urllib3.connectionpool][DEBUG   ][14199] \"GET /latest/meta-data/iam/security-credentials/ HTTP/1.1\" 200 9\r\n[requests.packages.urllib3.connectionpool][INFO    ][14199] Starting new HTTP connection (1): 169.254.169.254\r\n[requests.packages.urllib3.connectionpool][DEBUG   ][14199] \"GET /latest/meta-data/iam/security-credentials/PetroSalt HTTP/1.1\" 200 890\r\n[requests.packages.urllib3.connectionpool][INFO    ][14199] Starting new HTTP connection (1): 169.254.169.254\r\n[requests.packages.urllib3.connectionpool][DEBUG   ][14199] \"GET /latest/dynamic/instance-identity/document HTTP/1.1\" 200 429\r\n[salt.utils.s3    ][DEBUG   ][14199] S3 Request: https://petrocloud-releases.s3.amazonaws.com/?\r\n[salt.utils.s3    ][DEBUG   ][14199] S3 Headers::\r\n[salt.utils.s3    ][DEBUG   ][14199]     Authorization: AWS4-HMAC-SHA256 Credential=<redacted>, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=<redacted>\r\n[requests.packages.urllib3.connectionpool][INFO    ][14199] Starting new HTTPS connection (1): petrocloud-releases.s3.amazonaws.com\r\n[requests.packages.urllib3.connectionpool][DEBUG   ][14199] \"GET / HTTP/1.1\" 200 None\r\n[salt.utils.s3    ][DEBUG   ][14199] S3 Response Status Code: 200\r\n[salt.loaded.int.fileserver.s3fs][INFO    ][14199] s3_meta key: headers\r\n[salt.loaded.int.fileserver.s3fs][WARNING ][14199] S3 Error! Do you have any files in your S3 bucket? {}\r\n[salt.loaded.int.fileserver.s3fs][INFO    ][14199] Syncing local cache from S3...\r\n[salt.loaded.int.fileserver.s3fs][INFO    ][14199] Sync local cache from S3 completed.\r\n```\r\nNext I tried logging what was actually contained in the response from [utils/s3.py](https://github.com/saltstack/salt/blob/develop/salt/utils/s3.py#L237) and ended up with this:\r\n```\r\n[salt.loaded.int.fileserver.s3fs][INFO    ][2614] {'headers': ['Date', 'x-amz-id-2', 'Server', 'Transfer-Encoding', 'x-amz-request-id', 'x-amz-bucket-region', 'Content-Type']}\r\n```\r\nBy comparison, I get this from 2016.3.0 (truncated since it shows every file in the bucket):\r\n```\r\n[{'Name': '<redacted>'}, {'Prefix': None}, {'Marker': None}, {'MaxKeys': '1000'}, {'IsTruncated': 'false'}, {'LastModified': '<redacted>', 'ETag': '\"<redacted>\"', 'StorageClass': 'STANDARD', 'Key': 'base/', 'Owner': {'DisplayName': '<redacted>', 'ID': '<redacted>'}, 'Size': '0'}]\r\n```\r\n\r\n### Versions Report\r\n##### Master\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: 3.5.0\r\n       dateutil: 2.5.3\r\n          gitdb: 0.6.4\r\n      gitpython: 1.0.1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.8\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: 1.0.3\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.11+ (default, Apr 17 2016, 14:00:29)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 15.2.0\r\n           RAET: Not Installed\r\n          smmap: 0.9.0\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.1.4\r\n \r\nSystem Versions:\r\n           dist: Ubuntu 16.04 xenial\r\n        machine: x86_64\r\n        release: 4.4.0-22-generic\r\n         system: Linux\r\n        version: Ubuntu 16.04 xenial\r\n```\r\n##### Minion\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: 3.5.0\r\n       dateutil: 2.5.3\r\n          gitdb: 0.6.4\r\n      gitpython: 1.0.1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.8\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: 1.0.3\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.11+ (default, Apr 17 2016, 14:00:29)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 15.2.0\r\n           RAET: Not Installed\r\n          smmap: 0.9.0\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.1.4\r\n \r\nSystem Versions:\r\n           dist: Ubuntu 16.04 xenial\r\n        machine: x86_64\r\n        release: 4.4.0-22-generic\r\n         system: Linux\r\n        version: Ubuntu 16.04 xenial\r\n```"}}
{"actor":{"display_login":"gtmanfred"}}