{"action":"created","issue":{"url":"https://api.github.com/repos/ansible/ansible/issues/15603","repository_url":"https://api.github.com/repos/ansible/ansible","labels_url":"https://api.github.com/repos/ansible/ansible/issues/15603/labels{/name}","comments_url":"https://api.github.com/repos/ansible/ansible/issues/15603/comments","events_url":"https://api.github.com/repos/ansible/ansible/issues/15603/events","html_url":"https://github.com/ansible/ansible/issues/15603","id":151160626,"number":15603,"title":"Faulty parsing of JSON template/file lookups (aka JSON double-quotes to single-quotes conversion problem)","user":{"login":"sgnn7","id":175811,"avatar_url":"https://avatars.githubusercontent.com/u/175811?v=3","gravatar_id":"","url":"https://api.github.com/users/sgnn7","html_url":"https://github.com/sgnn7","followers_url":"https://api.github.com/users/sgnn7/followers","following_url":"https://api.github.com/users/sgnn7/following{/other_user}","gists_url":"https://api.github.com/users/sgnn7/gists{/gist_id}","starred_url":"https://api.github.com/users/sgnn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sgnn7/subscriptions","organizations_url":"https://api.github.com/users/sgnn7/orgs","repos_url":"https://api.github.com/users/sgnn7/repos","events_url":"https://api.github.com/users/sgnn7/events{/privacy}","received_events_url":"https://api.github.com/users/sgnn7/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ansible/ansible/labels/bug_report","name":"bug_report","color":"fbca04"},{"url":"https://api.github.com/repos/ansible/ansible/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"bcoca","id":836171,"avatar_url":"https://avatars.githubusercontent.com/u/836171?v=3","gravatar_id":"","url":"https://api.github.com/users/bcoca","html_url":"https://github.com/bcoca","followers_url":"https://api.github.com/users/bcoca/followers","following_url":"https://api.github.com/users/bcoca/following{/other_user}","gists_url":"https://api.github.com/users/bcoca/gists{/gist_id}","starred_url":"https://api.github.com/users/bcoca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcoca/subscriptions","organizations_url":"https://api.github.com/users/bcoca/orgs","repos_url":"https://api.github.com/users/bcoca/repos","events_url":"https://api.github.com/users/bcoca/events{/privacy}","received_events_url":"https://api.github.com/users/bcoca/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-04-26T15:08:31Z","updated_at":"2016-04-28T21:54:50Z","closed_at":null,"body":"##### ISSUE TYPE\r\n - Bug Report\r\n\r\n##### ANSIBLE VERSION\r\n```\r\n2.1.0\r\n```\r\n\r\n##### CONFIGURATION\r\n\r\nDefault from Git master\r\n\r\n##### OS / ENVIRONMENT\r\n\r\nDebian Jessie x86_64\r\n\r\n##### SUMMARY\r\nJSON `lookup('template', foobar)` content gets returned as variable that gets reparsed as dict (afaik), which somewhere gets serialized to string before it gets to the module and converts valid-JSON double-quotes to single-quotes so services or modules that expect valid JSON will fail with such data.\r\n\r\nThe problem is that Jinja2 templating is done recursively and I could not find any combination of `convert_*` arguments, Jinja2 filters, nor `#jinja2` overrides that would return the correct JSON from the template to the module and the classic override of putting a space before the string start isn't allowed when contacting AWS services. I believe the same issue exists with `lookup('file')` and any other lookup that would return JSON.\r\n\r\n<br />\r\n\r\nProblem code:\r\n- https://github.com/ansible/ansible/blob/devel/lib/ansible/template/__init__.py#L333-L335\r\n - Returned JSON from template ends up being templated again since it starts with `{` and returns a dict.\r\n\r\n<br/>\r\n- Bug fix PR that was fixing the symptom in `iam_policy` module with `policy_json` arg: https://github.com/ansible/ansible-modules-core/pull/3019\r\n\r\n- Another new (unreported yet) bug caused by this logic breaks `policy` argument in `s3_bucket` module here if you use a lookup: https://github.com/ansible/ansible-modules-extras/blob/devel/cloud/amazon/s3_bucket.py#L211\r\n\r\nI'd probably fix this myself but I just don't have the time to do it :(\r\n\r\n##### STEPS TO REPRODUCE\r\n```\r\n{\r\n  \"Version\": \"2012-10-17\",\r\n  \"Statement\": [\r\n    {\r\n      \"Sid\": \"AddPerm\",\r\n      \"Effect\": \"Allow\",\r\n      \"Principal\": \"*\",\r\n      \"Action\": \"s3:GetObject\",\r\n      \"Resource\": \"arn:aws:s3:::{{ item }}\"\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\n```\r\n---\r\n\r\n- hosts: all\r\n  connection: local\r\n  gather_facts: False\r\n\r\n  tasks:\r\n    - name: Setup S3 Bucket\r\n      s3_bucket:\r\n        name: \"{{ item }}\"\r\n        region: \"us-west-2\"\r\n        policy: \"{{ lookup('template','./bucket-public.json.j2', convert_data=False) }}\"\r\n        with_item:\r\n          - some_bucket_name\r\n```\r\n\r\n##### EXPECTED RESULTS\r\nJSON posted to AWS is valid and bucket is created\r\n\r\n##### ACTUAL RESULTS\r\n```\r\nPLAYBOOK: deploy.yml ***********************************************************\r\n1 plays in deploy.yml\r\n\r\nPLAY [all] *********************************************************************\r\n\r\nTASK [Setup S3 Bucket] *********************************************************\r\ntask path: /home/username/checkout/git-repo-name/deploy.yml:12\r\nFile lookup using /home/username/checkout/git-repo-name/bucket-public.json.j2 as file\r\n<localhost> ESTABLISH LOCAL CONNECTION FOR USER: username\r\n<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1461682432.3-159666311309506 `\" && echo \"` echo $HOME/.ansible/tmp/ansible-tmp-1461682432.3-159666311309506 `\" )'\r\n<localhost> PUT /tmp/tmph4kIsC TO /home/username/.ansible/tmp/ansible-tmp-1461682432.3-159666311309506/s3_bucket\r\n<localhost> EXEC /bin/sh -c 'LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python /home/username/.ansible/tmp/ansible-tmp-1461682432.3-159666311309506/s3_bucket; rm -rf \"/home/username/.ansible/tmp/ansible-tmp-1461682432.3-159666311309506/\" > /dev/null 2>&1'\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"failed\": true, \"invocation\": {\"module_args\": {\"aws_access_key\": null, \"aws_secret_key\": null, \"ec2_url\": null, \"force\": false, \"name\": \"some_bucket_name\", \"policy\": \"{'Version': '2012-10-17', 'Statement': [{'Action': 's3:GetObject', 'Principal': '*', 'Resource': 'arn:aws:s3:::some_bucket_name', 'Effect': 'Allow', 'Sid': 'AddPerm'}]}\", \"profile\": null, \"region\": \"us-west-2\", \"requester_pays\": false, \"s3_url\": null, \"security_token\": null, \"state\": \"present\", \"tags\": {}, \"validate_certs\": true, \"versioning\": false}, \"module_name\": \"s3_bucket\"}, \"msg\": \"Policies must be valid JSON and the first byte must be '{'\"}\r\n\r\n\r\nNO MORE HOSTS LEFT *************************************************************\r\n\tto retry, use: --limit @deploy.retry\r\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=0    changed=0    unreachable=0    failed=1\r\n```"},"comment":{"url":"https://api.github.com/repos/ansible/ansible/issues/comments/215574456","html_url":"https://github.com/ansible/ansible/issues/15603#issuecomment-215574456","issue_url":"https://api.github.com/repos/ansible/ansible/issues/15603","id":215574456,"user":{"login":"bcoca","id":836171,"avatar_url":"https://avatars.githubusercontent.com/u/836171?v=3","gravatar_id":"","url":"https://api.github.com/users/bcoca","html_url":"https://github.com/bcoca","followers_url":"https://api.github.com/users/bcoca/followers","following_url":"https://api.github.com/users/bcoca/following{/other_user}","gists_url":"https://api.github.com/users/bcoca/gists{/gist_id}","starred_url":"https://api.github.com/users/bcoca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcoca/subscriptions","organizations_url":"https://api.github.com/users/bcoca/orgs","repos_url":"https://api.github.com/users/bcoca/repos","events_url":"https://api.github.com/users/bcoca/events{/privacy}","received_events_url":"https://api.github.com/users/bcoca/received_events","type":"User","site_admin":false},"created_at":"2016-04-28T21:54:50Z","updated_at":"2016-04-28T21:54:50Z","body":"@sgnn7 the problem seems a bad assumption or bad documentation on what modules take, if you look at my fix I just changed the module to deal with whatever is passed in."}}