{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27194","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27194/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27194/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27194/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27194","id":159700661,"number":27194,"title":"cAdvisor /stats/summary endpoint in kubelet returns incorrect cpu usage numbers","user":{"login":"hasbro17","id":6660644,"avatar_url":"https://avatars.githubusercontent.com/u/6660644?v=3","gravatar_id":"","url":"https://api.github.com/users/hasbro17","html_url":"https://github.com/hasbro17","followers_url":"https://api.github.com/users/hasbro17/followers","following_url":"https://api.github.com/users/hasbro17/following{/other_user}","gists_url":"https://api.github.com/users/hasbro17/gists{/gist_id}","starred_url":"https://api.github.com/users/hasbro17/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hasbro17/subscriptions","organizations_url":"https://api.github.com/users/hasbro17/orgs","repos_url":"https://api.github.com/users/hasbro17/repos","events_url":"https://api.github.com/users/hasbro17/events{/privacy}","received_events_url":"https://api.github.com/users/hasbro17/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"Random-Liu","id":5821883,"avatar_url":"https://avatars.githubusercontent.com/u/5821883?v=3","gravatar_id":"","url":"https://api.github.com/users/Random-Liu","html_url":"https://github.com/Random-Liu","followers_url":"https://api.github.com/users/Random-Liu/followers","following_url":"https://api.github.com/users/Random-Liu/following{/other_user}","gists_url":"https://api.github.com/users/Random-Liu/gists{/gist_id}","starred_url":"https://api.github.com/users/Random-Liu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Random-Liu/subscriptions","organizations_url":"https://api.github.com/users/Random-Liu/orgs","repos_url":"https://api.github.com/users/Random-Liu/repos","events_url":"https://api.github.com/users/Random-Liu/events{/privacy}","received_events_url":"https://api.github.com/users/Random-Liu/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.3","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18/labels","id":1638460,"number":18,"title":"v1.3","description":"https://github.com/kubernetes/kubernetes/wiki/Release-1.3","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":107,"closed_issues":787,"state":"open","created_at":"2016-03-11T07:40:20Z","updated_at":"2016-06-18T01:48:33Z","due_on":"2016-06-24T07:00:00Z","closed_at":null},"comments":26,"created_at":"2016-06-10T19:01:23Z","updated_at":"2016-06-18T01:48:33Z","closed_at":"2016-06-18T01:48:33Z","body":"### Environment\r\nKubernetes version: 1.2.3\r\nDocker version: 1.10.3\r\n3 node(c4.xLarge) cluster on AWS running CoreOS 1010.4.0.\r\n### Issue\r\nAfter facing an issue with incorrect metrics being reported by heapster kubernetes/heapster#1177 I tried querying the cadvisor /stats/summary endpoint directly to see if that would give me consistent values for node cpu usage.\r\nI have one pod with cpu request=1000m and limit=1000m. In that pod I run a busy loop to consume a 100% of the cpu. This is what top shows on the node.\r\n<img width=\"648\" alt=\"screen shot 2016-06-10 at 10 07 52 am\" src=\"https://cloud.githubusercontent.com/assets/6660644/15975336/e922fe28-2f01-11e6-9dd9-7482fa52b4e4.png\">\r\n\r\nI query the /stats/summary endpoint every 5 seconds, however it seems that the latest timestamps are only updated every 15 seconds or so. Checking the `summary.Node.CPU.UsageNanoCores` value from the summary returned gives me the following output(formatted):\r\n\r\n```\r\nTS:2016-06-09T20:05:20-07:00, Percentage:105.539566 Val:1055395658\r\nTS:2016-06-09T20:05:31-07:00, Percentage:107.416097 Val:1074160974\r\nTS:2016-06-09T20:05:44-07:00, Percentage:108.195877 Val:1081958770\r\nTS:2016-06-09T20:05:59-07:00, Percentage:106.670910 Val:1066709101\r\nTS:2016-06-09T20:06:19-07:00, Percentage:14.360576 Val:143605762\r\nTS:2016-06-09T20:06:31-07:00, Percentage:108.000277 Val:1080002769\r\nTS:2016-06-09T20:06:41-07:00, Percentage:108.373232 Val:1083732315\r\nTS:2016-06-09T20:06:56-07:00, Percentage:107.025070 Val:1070250700\r\nTS:2016-06-09T20:07:16-07:00, Percentage:13.004869 Val:130048687\r\nTS:2016-06-09T20:07:31-07:00, Percentage:106.839146 Val:1068391461\r\nTS:2016-06-09T20:07:48-07:00, Percentage:107.614464 Val:1076144640\r\nTS:2016-06-09T20:08:06-07:00, Percentage:4.232330 Val:42323305\r\nTS:2016-06-09T20:08:20-07:00, Percentage:106.009173 Val:1060091732\r\nTS:2016-06-09T20:08:35-07:00, Percentage:108.121440 Val:1081214401\r\nTS:2016-06-09T20:08:50-07:00, Percentage:106.659561 Val:1066595609\r\nTS:2016-06-09T20:09:07-07:00, Percentage:1.724644 Val:17246439\r\nTS:2016-06-09T20:09:19-07:00, Percentage:106.633227 Val:1066332268\r\nTS:2016-06-09T20:09:38-07:00, Percentage:9.938621 Val:99386209\r\nTS:2016-06-09T20:09:53-07:00, Percentage:107.046112 Val:1070461118\r\nTS:2016-06-09T20:10:10-07:00, Percentage:3.373636 Val:33736361\r\nTS:2016-06-09T20:10:25-07:00, Percentage:107.338541 Val:1073385413\r\nTS:2016-06-09T20:10:39-07:00, Percentage:108.575783 Val:1085757834\r\nTS:2016-06-09T20:10:54-07:00, Percentage:107.055382 Val:1070553817\r\nTS:2016-06-09T20:11:13-07:00, Percentage:7.869509 Val:78695088\r\nTS:2016-06-09T20:11:32-07:00, Percentage:11.476262 Val:114762620\r\nTS:2016-06-09T20:11:45-07:00, Percentage:106.928681 Val:1069286811\r\nTS:2016-06-09T20:12:03-07:00, Percentage:3.309632 Val:33096320\r\nTS:2016-06-09T20:12:15-07:00, Percentage:105.832345 Val:1058323450\r\nTS:2016-06-09T20:12:34-07:00, Percentage:5.079409 Val:50794090\r\nTS:2016-06-09T20:12:47-07:00, Percentage:106.305439 Val:1063054389\r\nTS:2016-06-09T20:13:05-07:00, Percentage:3.613690 Val:36136900\r\nTS:2016-06-09T20:13:24-07:00, Percentage:9.785441 Val:97854409\r\nTS:2016-06-09T20:13:43-07:00, Percentage:12.661783 Val:126617830\r\n```\r\nAs you can see I'm not getting a steady report of near 100% cpu usage values for UsageNanoCores. Any idea why this might be the case or how I can debug this issue. Also is there any way I can change the resolution of the summary stats to get more fine grained reporting."}}
{"actor":{"display_login":"k8s-merge-robot"}}