{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/security_monkey/issues/290","labels_url":"https://api.github.com/repos/Netflix/security_monkey/issues/290/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/security_monkey/issues/290/comments","events_url":"https://api.github.com/repos/Netflix/security_monkey/issues/290/events","html_url":"https://github.com/Netflix/security_monkey/issues/290","id":126763123,"number":290,"title":"botocore/Boto 3 calls not getting proxy settings","user":{"login":"finalriposte","id":7147968,"avatar_url":"https://avatars.githubusercontent.com/u/7147968?v=3","gravatar_id":"","url":"https://api.github.com/users/finalriposte","html_url":"https://github.com/finalriposte","followers_url":"https://api.github.com/users/finalriposte/followers","following_url":"https://api.github.com/users/finalriposte/following{/other_user}","gists_url":"https://api.github.com/users/finalriposte/gists{/gist_id}","starred_url":"https://api.github.com/users/finalriposte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/finalriposte/subscriptions","organizations_url":"https://api.github.com/users/finalriposte/orgs","repos_url":"https://api.github.com/users/finalriposte/repos","events_url":"https://api.github.com/users/finalriposte/events{/privacy}","received_events_url":"https://api.github.com/users/finalriposte/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-14T22:29:27Z","updated_at":"2016-01-20T04:45:35Z","closed_at":null,"body":"I have an Ubuntu and a CentOS instance that has issues with botocore calls.\r\nAll boto2 calls are fine.\r\nBotocore calls are not getting the proxy settings.\r\nExamples of the log messages are:\r\nsecurity_monkey-deploy.log:51326:2016-01-15 08:04:11,143 DEBUG: Adding ('iamrole', u'<ACCOUNT>', 'universal') to the exceptions list. Exception was: \"Problem Connecting to iamrole/<ACCOUNT>/None:\\nHTTPSConneconPool(host='iam.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7fa6880f7710>, 'Connection to iam.amazonawcom timed out. (connect timeout=60)'))\" [in /usr/local/src/security_monkey/security_monkey/watcher.py:161]\r\n\r\nsecurity_monkey-deploy.log:53229:2016-01-15 09:19:15,516 DEBUG: Adding ('elb', u'<ACCOUNT>', u'ap-southeast-2', u'<ELB_ID>') to the exceptions list. Exception was: HTTPSConnectionPool(host='elasticloadbalancing.ap-southeast-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7fa689cdf690>, 'Connection to elasticloadbalancing.ap-southeast-2.amazonaws.com timed out. (connect timeout=60)')) [in /usr/local/src/security_monkey/security_monkey/watcher.py:161]\r\n\r\nNetstat -an shows that these calls are trying to go straight to the API instead of the proxy:\r\ntcp        0      1 10.x.x.x:48739     54.240.193.98:443       SYN_SENT\r\n\r\nThe proxies are set as part of the environment profile.\r\n\r\nBoto versions are:\r\nboto (2.38.0)\r\nboto3 (1.2.3)\r\nbotocore (1.3.17)\r\npython --version\r\nPython 2.7.5\r\n\r\nurllib3 version\r\nurllib3 (1.14)\r\n\r\nHas anyone else seen this and found a resolution?\r\nI could setup squid and transparently proxy everything, but I'd really prefer to see why the botocore calls are failing so if anyone else has the problem it is documented - as I suspect I am overlooking something simple."},"comment":{"url":"https://api.github.com/repos/Netflix/security_monkey/issues/comments/173085881","html_url":"https://github.com/Netflix/security_monkey/issues/290#issuecomment-173085881","issue_url":"https://api.github.com/repos/Netflix/security_monkey/issues/290","id":173085881,"user":{"login":"monkeysecurity","id":8009126,"avatar_url":"https://avatars.githubusercontent.com/u/8009126?v=3","gravatar_id":"","url":"https://api.github.com/users/monkeysecurity","html_url":"https://github.com/monkeysecurity","followers_url":"https://api.github.com/users/monkeysecurity/followers","following_url":"https://api.github.com/users/monkeysecurity/following{/other_user}","gists_url":"https://api.github.com/users/monkeysecurity/gists{/gist_id}","starred_url":"https://api.github.com/users/monkeysecurity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monkeysecurity/subscriptions","organizations_url":"https://api.github.com/users/monkeysecurity/orgs","repos_url":"https://api.github.com/users/monkeysecurity/repos","events_url":"https://api.github.com/users/monkeysecurity/events{/privacy}","received_events_url":"https://api.github.com/users/monkeysecurity/received_events","type":"User","site_admin":false},"created_at":"2016-01-20T04:45:35Z","updated_at":"2016-01-20T04:45:35Z","body":"https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L272\r\n\r\n```\r\n    def _get_proxies(self, url):\r\n        # We could also support getting proxies from a config file,\r\n        # but for now proxy support is taken from the environment.\r\n        return get_environ_proxies(url)\r\n```\r\n\r\nIt looks like botocore may be relying on urllib to pull proxy information out of environment variables and does not accept proxy settings from the boto config file.\r\n\r\nhttps://docs.python.org/3/library/urllib.request.html#urllib.request.getproxies\r\n\r\n> urllib.request.getproxies()\r\n> This helper function returns a dictionary of scheme to proxy server URL mappings. It scans the environment for variables named <scheme>_proxy, in a case insensitive approach,\r\n\r\nI would think you could set a https_proxy environment variable and it would work.  Hopefully http_proxy wouldn't also be required.\r\n\r\nLet us know if this works."}}