{"action":"created","issue":{"url":"https://api.github.com/repos/DataDog/integrations-core/issues/1346","repository_url":"https://api.github.com/repos/DataDog/integrations-core","labels_url":"https://api.github.com/repos/DataDog/integrations-core/issues/1346/labels{/name}","comments_url":"https://api.github.com/repos/DataDog/integrations-core/issues/1346/comments","events_url":"https://api.github.com/repos/DataDog/integrations-core/issues/1346/events","html_url":"https://github.com/DataDog/integrations-core/issues/1346","id":311472119,"number":1346,"title":"kube-state-metrics timing out with agent v6.1.0","user":{"login":"c-knowles","id":3053013,"avatar_url":"https://avatars3.githubusercontent.com/u/3053013?v=4","gravatar_id":"","url":"https://api.github.com/users/c-knowles","html_url":"https://github.com/c-knowles","followers_url":"https://api.github.com/users/c-knowles/followers","following_url":"https://api.github.com/users/c-knowles/following{/other_user}","gists_url":"https://api.github.com/users/c-knowles/gists{/gist_id}","starred_url":"https://api.github.com/users/c-knowles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-knowles/subscriptions","organizations_url":"https://api.github.com/users/c-knowles/orgs","repos_url":"https://api.github.com/users/c-knowles/repos","events_url":"https://api.github.com/users/c-knowles/events{/privacy}","received_events_url":"https://api.github.com/users/c-knowles/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-04-05T04:52:13Z","updated_at":"2018-04-12T09:41:20Z","closed_at":null,"author_association":"NONE","body":"We've just switched from agent v5 using our own YAML to v6.1.0 using the Helm chart and we appear to have hit the same problem as https://github.com/kubernetes/charts/issues/1466. We deploy kube-state-metrics separately rather than bundle it as a helm dependency. For that we're on 1.2.0.\r\n\r\nAfter the upgrade, most things are working ok but kube state metrics are not coming through any more when non of the rest of the system has changed.\r\n\r\nThis is what we see in the logs:\r\n\r\n```\r\ndatadog-master-rcd8v datadog [ AGENT ] 2018-04-05 04:33:04 UTC | ERROR | (runner.go:276 in work) | Error running check kubernetes_state: [{\"message\": \"HTTPConnectionPool(host='10.2.13.182', port=8080): Read timed out. (read timeout=1)\", \"traceback\": \"Traceback (most recent call last):\\n  File \\\"/opt/datadog-agent/bin/agent/dist/checks/__init__.py\\\", line 332, in run\\n    self.check(copy.deepcopy(self.instances[0]))\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubernetes_state/kubernetes_state.py\\\", line 196, in check\\n    self.process(endpoint, send_histograms_buckets=send_buckets, instance=instance)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/mixins.py\\\", line 350, in process\\n    for metric in self.scrape_metrics(endpoint):\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/mixins.py\\\", line 314, in scrape_metrics\\n    response = self.poll(endpoint)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/mixins.py\\\", line 467, in poll\\n    response = requests.get(endpoint, headers=headers, stream=True, timeout=1, cert=cert, verify=verify)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py\\\", line 72, in get\\n    return request('get', url, params=params, **kwargs)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py\\\", line 58, in request\\n    return session.request(method=method, url=url, **kwargs)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py\\\", line 508, in request\\n    resp = self.send(prep, **send_kwargs)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py\\\", line 618, in send\\n    r = adapter.send(request, **kwargs)\\n  File \\\"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/adapters.py\\\", line 521, in send\\n    raise ReadTimeout(e, request=request)\\nReadTimeout: HTTPConnectionPool(host='10.2.13.182', port=8080): Read timed out. (read timeout=1)\\n\"}]\r\n```\r\n\r\nI've confirmed `10.2.13.182` is the correct pod IP address; as mentioned none of that side of things has changed so the AD seems to still be working.\r\n\r\nI went to the datadog pod in question and did a little investigation on the timing:\r\n```\r\ntime curl -v --silent --output /dev/null --show-error --fail 10.2.13.182:8080/metrics\r\n```\r\n\r\nThe curl responds fine with content but I've never got it to go below 1 second. However, it appears 1 second is hard coded:\r\nhttps://github.com/DataDog/integrations-core/blob/69e3a575ff5d9dc62703b9b9f7789d98e91a2ec5/datadog-checks-base/datadog_checks/checks/prometheus/mixins.py#L472\r\n\r\nThe payload is ~11214 bytes on that cluster. I tried increasing resources to the kube state metrics pod but wasn't able to get it to go any faster - the stats for the kube-state-metrics container show it's not even using close to its allocated cpu/memory.\r\n\r\nI'm not sure how best to deal with this, as a start the timeout could be configurable but perhaps there are some overall plans to deal with this in some better wat."},"comment":{"url":"https://api.github.com/repos/DataDog/integrations-core/issues/comments/380742362","html_url":"https://github.com/DataDog/integrations-core/issues/1346#issuecomment-380742362","issue_url":"https://api.github.com/repos/DataDog/integrations-core/issues/1346","id":380742362,"user":{"login":"antoinepouille","id":25852569,"avatar_url":"https://avatars0.githubusercontent.com/u/25852569?v=4","gravatar_id":"","url":"https://api.github.com/users/antoinepouille","html_url":"https://github.com/antoinepouille","followers_url":"https://api.github.com/users/antoinepouille/followers","following_url":"https://api.github.com/users/antoinepouille/following{/other_user}","gists_url":"https://api.github.com/users/antoinepouille/gists{/gist_id}","starred_url":"https://api.github.com/users/antoinepouille/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antoinepouille/subscriptions","organizations_url":"https://api.github.com/users/antoinepouille/orgs","repos_url":"https://api.github.com/users/antoinepouille/repos","events_url":"https://api.github.com/users/antoinepouille/events{/privacy}","received_events_url":"https://api.github.com/users/antoinepouille/received_events","type":"User","site_admin":false},"created_at":"2018-04-12T09:41:20Z","updated_at":"2018-04-12T09:41:20Z","author_association":"MEMBER","body":"@c-knowles Thanks for opening this issue! We will review to increase the value of the timeout here. The details you sent are most helpful in that end.\r\nWe will also get back to you on the support case we have in parallel, thanks for your patience again here!"}}
{"actor":{"display_login":"antoinepouille"}}