{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/766","repository_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos","labels_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/766/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/766/comments","events_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/766/events","html_url":"https://github.com/mesosphere/kubernetes-mesos/issues/766","id":131491701,"number":766,"title":"cadvisor docker stats not working in v0.7.2-v1.15 release","user":{"login":"ravilr","id":3928411,"avatar_url":"https://avatars.githubusercontent.com/u/3928411?v=3","gravatar_id":"","url":"https://api.github.com/users/ravilr","html_url":"https://github.com/ravilr","followers_url":"https://api.github.com/users/ravilr/followers","following_url":"https://api.github.com/users/ravilr/following{/other_user}","gists_url":"https://api.github.com/users/ravilr/gists{/gist_id}","starred_url":"https://api.github.com/users/ravilr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ravilr/subscriptions","organizations_url":"https://api.github.com/users/ravilr/orgs","repos_url":"https://api.github.com/users/ravilr/repos","events_url":"https://api.github.com/users/ravilr/events{/privacy}","received_events_url":"https://api.github.com/users/ravilr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/labels/area/executor","name":"area/executor","color":"60bdff"},{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/labels/class/bug","name":"class/bug","color":"fc2929"},{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/labels/integration/cadvisor","name":"integration/cadvisor","color":"dbaa00"},{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/labels/integration/k8s","name":"integration/k8s","color":"fbca04"},{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/labels/priority/P2","name":"priority/P2","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/milestones/36","html_url":"https://github.com/mesosphere/kubernetes-mesos/milestones/v0.7.3","labels_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/milestones/36/labels","id":1546928,"number":36,"title":"v0.7.3","description":"","creator":{"login":"jdef","id":2348332,"avatar_url":"https://avatars.githubusercontent.com/u/2348332?v=3","gravatar_id":"","url":"https://api.github.com/users/jdef","html_url":"https://github.com/jdef","followers_url":"https://api.github.com/users/jdef/followers","following_url":"https://api.github.com/users/jdef/following{/other_user}","gists_url":"https://api.github.com/users/jdef/gists{/gist_id}","starred_url":"https://api.github.com/users/jdef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdef/subscriptions","organizations_url":"https://api.github.com/users/jdef/orgs","repos_url":"https://api.github.com/users/jdef/repos","events_url":"https://api.github.com/users/jdef/events{/privacy}","received_events_url":"https://api.github.com/users/jdef/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":5,"state":"open","created_at":"2016-01-27T18:47:49Z","updated_at":"2016-02-09T14:43:14Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2016-02-04T22:26:54Z","updated_at":"2016-02-09T20:28:46Z","closed_at":null,"body":"@jdef \r\nversion : v0.7.2-v1.1.5\r\n\r\nkubelet embedded cadvisor reporting docker container stats has stopped working since we upgraded to v0.7.2-v1.1.5 (from v0.7.0-v1.1.1). \r\n\r\nkubelet stats are exposed via http on port 10255.\r\nhttp://mesos-slave1.host.com:10255/stats/kube-system/kube-dns-v10-2-yrmw0/6d14f56e-ca21-11e5-a652-fa163eee7610/etcd\r\nreturns 'Internal Error: unable to find Docker container \"0df7cae297149a47684df7360b164329e40287f824cacfc161f671f8c38310bc' even though the container id exists in the system.\r\n\r\nhttp://mesos-slave1.host.com:4194/ (cadvisor ui) also doesn't list any docker containers in it UI. \r\n\r\nNot sure if this is an issue in the upstream or something to do with they way kubelet-executor launches kubelet. Any idea?\r\n\r\nheapster pod calls the above kubelet stats port for cluster stats aggregation, and because of this issue, heapster stats for pods are also empty. \r\n\r\nthis is where it is failing:\r\nhttps://github.com/mesosphere/kubernetes/blob/v0.7.2-v1.1.5/pkg/kubelet/server.go#L1022\r\nhttps://github.com/google/cadvisor/blob/v0.19.3.2/manager/manager.go#L526\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/comments/182047340","html_url":"https://github.com/mesosphere/kubernetes-mesos/issues/766#issuecomment-182047340","issue_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/766","id":182047340,"user":{"login":"ravilr","id":3928411,"avatar_url":"https://avatars.githubusercontent.com/u/3928411?v=3","gravatar_id":"","url":"https://api.github.com/users/ravilr","html_url":"https://github.com/ravilr","followers_url":"https://api.github.com/users/ravilr/followers","following_url":"https://api.github.com/users/ravilr/following{/other_user}","gists_url":"https://api.github.com/users/ravilr/gists{/gist_id}","starred_url":"https://api.github.com/users/ravilr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ravilr/subscriptions","organizations_url":"https://api.github.com/users/ravilr/orgs","repos_url":"https://api.github.com/users/ravilr/repos","events_url":"https://api.github.com/users/ravilr/events{/privacy}","received_events_url":"https://api.github.com/users/ravilr/received_events","type":"User","site_admin":false},"created_at":"2016-02-09T20:28:46Z","updated_at":"2016-02-09T20:28:46Z","body":"apparently, this turns out to be an upstream issue in cadvisor. there seems to be a change related to how cadvisor recognizes docker managed containers in systemd based systems that went in around cadvisor 0.19.{1-3} release :\r\nhttps://github.com/google/cadvisor/pull/821\r\nthe above change seems to have broken use cases where docker containers are launched with --cgroup_parent, which the mesos executor does. \r\n\r\nadding --nosystemd=true flag to the kubelet executor appears to bring back the cadvisor docker container stats working along with the heapster stats(atleast on RHEL system)."}}