{"action":"created","issue":{"url":"https://api.github.com/repos/grafana/kubernetes-app/issues/44","repository_url":"https://api.github.com/repos/grafana/kubernetes-app","labels_url":"https://api.github.com/repos/grafana/kubernetes-app/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/grafana/kubernetes-app/issues/44/comments","events_url":"https://api.github.com/repos/grafana/kubernetes-app/issues/44/events","html_url":"https://github.com/grafana/kubernetes-app/issues/44","id":362364227,"node_id":"MDU6SXNzdWUzNjIzNjQyMjc=","number":44,"title":"Container dashboard showing doubled metrics","user":{"login":"matheusneder","id":6011646,"node_id":"MDQ6VXNlcjYwMTE2NDY=","avatar_url":"https://avatars1.githubusercontent.com/u/6011646?v=4","gravatar_id":"","url":"https://api.github.com/users/matheusneder","html_url":"https://github.com/matheusneder","followers_url":"https://api.github.com/users/matheusneder/followers","following_url":"https://api.github.com/users/matheusneder/following{/other_user}","gists_url":"https://api.github.com/users/matheusneder/gists{/gist_id}","starred_url":"https://api.github.com/users/matheusneder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matheusneder/subscriptions","organizations_url":"https://api.github.com/users/matheusneder/orgs","repos_url":"https://api.github.com/users/matheusneder/repos","events_url":"https://api.github.com/users/matheusneder/events{/privacy}","received_events_url":"https://api.github.com/users/matheusneder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-20T21:00:09Z","updated_at":"2018-10-29T14:51:38Z","closed_at":null,"author_association":"NONE","body":"Hi!\r\n\r\nI have a kubernetes cluster on which I recently installed prometheus, through prometheus operator, following official prometheus guide.\r\n\r\nI connected **Grafana Kubernetes App** to kubernetes/prometheus then I noticed that the metrics exposed at K8s container dashboard for memory and cpu was been showing doubled.\r\n\r\nAfter some debug, I discovered the bellow query for cpu usage:\r\n\r\n```\r\nsum(irate(container_cpu_usage_seconds_total{pod_name=~\"my-pod\"}[2m])) by (pod_name)\r\n```\r\n\r\nExecuting the same query but removing aggregate (`container_cpu_usage_seconds_total{pod_name=~\"my-pod\"}`) I could see that there is an extra item that represents the sum of all containers in POD as in table bellow:\r\n\r\n|Element |Value|\r\n|--|--|\r\n|container_cpu_usage_seconds_total{container_name=\"POD\",cpu=\"total\",endpoint=\"http-metrics\",id=\"/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990/da0ebc2deef52414bc2f1cc8cb874956b6c3dc478ea8df0d2eb3106a605ba21c\",image=\"k8s.gcr.io/pause:3.1\",instance=\"10.17.14.237:10255\",job=\"kubelet\",name=\"k8s_POD_my-pod-b54bc7874-bpmhd_pocs_1bb36063-bcf4-11e8-bba4-00505684c990_0\",namespace=\"pocs\",pod_name=\"my-pod-b54bc7874-bpmhd\",service=\"kubelet\"}|0.012306132|\r\n|container_cpu_usage_seconds_total{container_name=\"my-pod-rev2\",cpu=\"total\",endpoint=\"http-metrics\",id=\"/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990/f712e91d11913f01daa863fd3e5d59179afb40a7f1e8ff67cf8f24874d7fd44c\",image=\"sha256:2f41732fe757bb8dc7739b5a536ea38ed3aa9c4bc8f6c3e3c342e4952b0e63e7\",instance=\"10.17.14.237:10255\",job=\"kubelet\",name=\"k8s_my-pod-rev2_my-pod-b54bc7874-bpmhd_pocs_1bb36063-bcf4-11e8-bba4-00505684c990_1\",namespace=\"pocs\",pod_name=\"my-pod-b54bc7874-bpmhd\",service=\"kubelet\"}|107.36438691|\r\n|container_cpu_usage_seconds_total{cpu=\"total\",endpoint=\"http-metrics\",id=\"/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990\",instance=\"10.17.14.237:10255\",job=\"kubelet\",namespace=\"pocs\",pod_name=\"my-pod-b54bc7874-bpmhd\",service=\"kubelet\"}|145.636093587|\r\n\r\nSo, this justify the doubled metrics.\r\n\r\nThe extra item with the sum of containers (the 3th item on the above table) don't contains the container_name attribute. \r\n\r\nIn order to avoid **Grafana Kubernetes App** showing doubled values, I edited  queries on dashboard definition by adding `container_name=~\".+\"` to filter, so the query for cpu_usage now looks as:\r\n\r\n```\r\nsum(irate(container_cpu_usage_seconds_total{pod_name=~\"my-pod\", container_name=~\".+\"}[2m])) by (pod_name)\r\n```\r\n\r\nI'm wondering if there is a better way to handle this. Have you ever seen this scenario before? \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/grafana/kubernetes-app/issues/comments/433939902","html_url":"https://github.com/grafana/kubernetes-app/issues/44#issuecomment-433939902","issue_url":"https://api.github.com/repos/grafana/kubernetes-app/issues/44","id":433939902,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzkzOTkwMg==","user":{"login":"pezzati","id":6584487,"node_id":"MDQ6VXNlcjY1ODQ0ODc=","avatar_url":"https://avatars0.githubusercontent.com/u/6584487?v=4","gravatar_id":"","url":"https://api.github.com/users/pezzati","html_url":"https://github.com/pezzati","followers_url":"https://api.github.com/users/pezzati/followers","following_url":"https://api.github.com/users/pezzati/following{/other_user}","gists_url":"https://api.github.com/users/pezzati/gists{/gist_id}","starred_url":"https://api.github.com/users/pezzati/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pezzati/subscriptions","organizations_url":"https://api.github.com/users/pezzati/orgs","repos_url":"https://api.github.com/users/pezzati/repos","events_url":"https://api.github.com/users/pezzati/events{/privacy}","received_events_url":"https://api.github.com/users/pezzati/received_events","type":"User","site_admin":false},"created_at":"2018-10-29T14:51:38Z","updated_at":"2018-10-29T14:51:38Z","author_association":"NONE","body":"ME TOO!!"}}
{"actor":{"display_login":"pezzati"}}