{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/elastalert/issues/551","repository_url":"https://api.github.com/repos/Yelp/elastalert","labels_url":"https://api.github.com/repos/Yelp/elastalert/issues/551/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/elastalert/issues/551/comments","events_url":"https://api.github.com/repos/Yelp/elastalert/issues/551/events","html_url":"https://github.com/Yelp/elastalert/issues/551","id":157073420,"number":551,"title":"AWS Elasticsearch keeps giving 403 unathorized","user":{"login":"GittyHup","id":4475946,"avatar_url":"https://avatars.githubusercontent.com/u/4475946?v=3","gravatar_id":"","url":"https://api.github.com/users/GittyHup","html_url":"https://github.com/GittyHup","followers_url":"https://api.github.com/users/GittyHup/followers","following_url":"https://api.github.com/users/GittyHup/following{/other_user}","gists_url":"https://api.github.com/users/GittyHup/gists{/gist_id}","starred_url":"https://api.github.com/users/GittyHup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GittyHup/subscriptions","organizations_url":"https://api.github.com/users/GittyHup/orgs","repos_url":"https://api.github.com/users/GittyHup/repos","events_url":"https://api.github.com/users/GittyHup/events{/privacy}","received_events_url":"https://api.github.com/users/GittyHup/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-05-26T20:36:20Z","updated_at":"2016-05-26T23:08:51Z","closed_at":null,"body":"Here is my config.yaml\r\n\r\n\t# This is the folder that contains the rule yaml files\r\n\t# Any .yaml file will be loaded as a rule\r\n\trules_folder: example_rules\r\n\r\n\t# How often ElastAlert will query elasticsearch\r\n\t# The unit can be anything from weeks to seconds\r\n\trun_every:\r\n\t  minutes: 1\r\n\r\n\t# ElastAlert will buffer results from the most recent\r\n\t# period of time, in case some log sources are not in real time\r\n\tbuffer_time:\r\n\t  minutes: 15\r\n\r\n\t# The elasticsearch hostname for metadata writeback\r\n\t# Note that every rule can have it's own elasticsearch host\r\n\tes_host: search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com\r\n\r\n\t# The elasticsearch port\r\n\tes_port: 80\r\n\r\n\taws_region: us-west-2\r\n\r\n\t#boto_profile: default\r\n\r\n\t# Optional URL prefix for elasticsearch\r\n\tes_url_prefix: elasticsearch\r\n\r\n\t# Connect with SSL to elasticsearch\r\n\t#use_ssl: True\r\n\r\n\t# Option basic-auth username and password for elasticsearch\r\n\t#es_username: someusername\r\n\t#es_password: somepassword\r\n\r\n\t# The index on es_host which is used for metadata storage\r\n\t# This can be a unmapped index, but it is recommended that you run\r\n\t# elastalert-create-index to set a mapping\r\n\twriteback_index: elastalert_status\r\n\r\n\t# If an alert fails for some reason, ElastAlert will retry\r\n\t# sending the alert until this time period has elapsed\r\n\talert_time_limit:\r\n\t  days: 2\r\n\r\n\r\nAnd the error that I get:\r\n\r\n\tWARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert/_search?size=1000 [status:403 request:0.015s]\r\n\tWARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:403 request:0.005s]\r\n\tERROR:root:Error querying for last run: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster\"}')\r\n\tWARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.003s]\r\n\tERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')\r\n\tTraceback (most recent call last):\r\n\t  File \"elastalert/elastalert.py\", line 998, in writeback\r\n\t\tdoc_type=doc_type, body=body)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py\", line 69, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py\", line 248, in create\r\n\t\treturn self.index(index, doc_type, body, id=id, params=params, op_type='create')\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py\", line 69, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py\", line 279, in index\r\n\t\t_make_path(index, doc_type, id), params=params, body=body)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/transport.py\", line 329, in perform_request\r\n\t\tstatus, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py\", line 109, in perform_request\r\n\t\tself._raise_error(response.status, raw_data)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py\", line 108, in _raise_error\r\n\t\traise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\r\n\tAuthorizationException: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')\r\n\tWARNING:elasticsearch:GET /logstash-*/_search?_source_include=%40timestamp%2C%2A&ignore_unavailable=true&size=10000 [status:403 request:0.006s]\r\n\tERROR:root:Error running query: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster\"}')\r\n\tWARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.004s]\r\n\tERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')\r\n\tTraceback (most recent call last):\r\n\t  File \"elastalert/elastalert.py\", line 998, in writeback\r\n\t\tdoc_type=doc_type, body=body)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py\", line 69, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py\", line 248, in create\r\n\t\treturn self.index(index, doc_type, body, id=id, params=params, op_type='create')\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py\", line 69, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py\", line 279, in index\r\n\t\t_make_path(index, doc_type, id), params=params, body=body)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/transport.py\", line 329, in perform_request\r\n\t\tstatus, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py\", line 109, in perform_request\r\n\t\tself._raise_error(response.status, raw_data)\r\n\t  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py\", line 108, in _raise_error\r\n\t\traise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\r\n\tAuthorizationException: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')\r\n\r\nMy IAM role attached to instance with elastalert gives full access to es (elasticsearch):\r\n\r\n\t{\r\n\t\t\"Version\": \"2012-10-17\",\r\n\t\t\"Statement\": [\r\n\t\t\t{\r\n\t\t\t\t\"Sid\": \"Stmt1454694581000\",\r\n\t\t\t\t\"Effect\": \"Allow\",\r\n\t\t\t\t\"Action\": [\r\n\t\t\t\t\t\"cloudwatch:*\"\r\n\t\t\t\t],\r\n\t\t\t\t\"Resource\": [\r\n\t\t\t\t\t\"*\"\r\n\t\t\t\t]\r\n\t\t\t},\r\n\t\t\t{\r\n\t\t\t\t\"Sid\": \"Stmt1454694609000\",\r\n\t\t\t\t\"Effect\": \"Allow\",\r\n\t\t\t\t\"Action\": [\r\n\t\t\t\t\t\"events:*\"\r\n\t\t\t\t],\r\n\t\t\t\t\"Resource\": [\r\n\t\t\t\t\t\"*\"\r\n\t\t\t\t]\r\n\t\t\t},\r\n\t\t\t{\r\n\t\t\t\t\"Sid\": \"Stmt1454694630000\",\r\n\t\t\t\t\"Effect\": \"Allow\",\r\n\t\t\t\t\"Action\": [\r\n\t\t\t\t\t\"logs:*\"\r\n\t\t\t\t],\r\n\t\t\t\t\"Resource\": [\r\n\t\t\t\t\t\"*\"\r\n\t\t\t\t]\r\n\t\t\t},\r\n\t\t\t{\r\n\t\t\t\t\"Sid\": \"Stmt1454694649000\",\r\n\t\t\t\t\"Effect\": \"Allow\",\r\n\t\t\t\t\"Action\": [\r\n\t\t\t\t\t\"es:*\"\r\n\t\t\t\t],\r\n\t\t\t\t\"Resource\": [\r\n\t\t\t\t\t\"*\"\r\n\t\t\t\t]\r\n\t\t\t}\r\n\t\t]\r\n\t}\r\n\r\nI confirmed with IAM policy simulator as well as other cloudformation instances that use other apps to access elasticsearch in AWS:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/4475946/15589206/b1e43a56-2346-11e6-913f-766e58629595.png)\r\n"},"comment":{"url":"https://api.github.com/repos/Yelp/elastalert/issues/comments/222020222","html_url":"https://github.com/Yelp/elastalert/issues/551#issuecomment-222020222","issue_url":"https://api.github.com/repos/Yelp/elastalert/issues/551","id":222020222,"user":{"login":"GittyHup","id":4475946,"avatar_url":"https://avatars.githubusercontent.com/u/4475946?v=3","gravatar_id":"","url":"https://api.github.com/users/GittyHup","html_url":"https://github.com/GittyHup","followers_url":"https://api.github.com/users/GittyHup/followers","following_url":"https://api.github.com/users/GittyHup/following{/other_user}","gists_url":"https://api.github.com/users/GittyHup/gists{/gist_id}","starred_url":"https://api.github.com/users/GittyHup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GittyHup/subscriptions","organizations_url":"https://api.github.com/users/GittyHup/orgs","repos_url":"https://api.github.com/users/GittyHup/repos","events_url":"https://api.github.com/users/GittyHup/events{/privacy}","received_events_url":"https://api.github.com/users/GittyHup/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T23:08:51Z","updated_at":"2016-05-26T23:08:51Z","body":"I printed out the aws parameters in the auth file, but I still get error:\r\n\r\nSomething about \"AttributeError: 'AWSRequestsAuth' object has no attribute 'split'\"\r\n\r\n\t---------------------------------------------------\r\n\t[ACCESS_KEY]\r\n\t[TOKEN_HIDDEN]\r\n\tsearch-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com\r\n\tus-west-2\r\n\t---------------------------------------------------\r\n\tERROR:root:Traceback (most recent call last):\r\n\t  File \"elastalert/elastalert.py\", line 753, in run_all_rules\r\n\t\tnum_matches = self.run_rule(rule, endtime, self.starttime)\r\n\t  File \"elastalert/elastalert.py\", line 516, in run_rule\r\n\t\tself.current_es = self.new_elasticsearch(rule_es_conn_config)\r\n\t  File \"elastalert/elastalert.py\", line 143, in new_elasticsearch\r\n\t\tsend_get_body_as=es_conn_conf['send_get_body_as'])\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 150, in __init__\r\n\t\tself.transport = transport_class(_normalize_hosts(hosts), **kwargs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 113, in __init__\r\n\t\tself.set_connections(hosts)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 166, in set_connections\r\n\t\tconnections = map(_create_connection, hosts)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 165, in _create_connection\r\n\t\treturn self.connection_class(**kwargs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py\", line 35, in __init__\r\n\t\thttp_auth = http_auth.split(':', 1)\r\n\tAttributeError: 'AWSRequestsAuth' object has no attribute 'split'\r\n\r\n\tERROR:root:Uncaught exception running rule Example rule: 'AWSRequestsAuth' object has no attribute 'split'\r\n\tWARNING:elasticsearch:POST http://search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com:80/elastalert_status/elastalert_error?op_type=create [status:403 request:0.007s]\r\n\tERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')\r\n\tTraceback (most recent call last):\r\n\t  File \"elastalert/elastalert.py\", line 1022, in writeback\r\n\t\tdoc_type=doc_type, body=body)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 68, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 227, in create\r\n\t\treturn self.index(index, doc_type, body, id=id, params=params, op_type='create')\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 68, in _wrapped\r\n\t\treturn func(*args, params=params, **kwargs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 257, in index\r\n\t\t_make_path(index, doc_type, id), params=params, body=body)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 301, in perform_request\r\n\t\tstatus, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py\", line 72, in perform_request\r\n\t\tself._raise_error(response.status_code, raw_data)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py\", line 102, in _raise_error\r\n\t\traise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\r\n\tAuthorizationException: TransportError(403, u'{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster\"}')"}}
{"actor":{"display_login":"GittyHup"}}