{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/26000","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/26000/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/26000/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/26000/events","html_url":"https://github.com/kubernetes/kubernetes/issues/26000","id":156065049,"number":26000,"title":"kubelet Image garbage collection failed: unable to find data for container /","user":{"login":"azell","id":489196,"avatar_url":"https://avatars.githubusercontent.com/u/489196?v=3","gravatar_id":"","url":"https://api.github.com/users/azell","html_url":"https://github.com/azell","followers_url":"https://api.github.com/users/azell/followers","following_url":"https://api.github.com/users/azell/following{/other_user}","gists_url":"https://api.github.com/users/azell/gists{/gist_id}","starred_url":"https://api.github.com/users/azell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azell/subscriptions","organizations_url":"https://api.github.com/users/azell/orgs","repos_url":"https://api.github.com/users/azell/repos","events_url":"https://api.github.com/users/azell/events{/privacy}","received_events_url":"https://api.github.com/users/azell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-20T22:52:15Z","updated_at":"2016-05-20T22:52:15Z","closed_at":null,"body":"Cluster 1.2.2 settings:\r\n\r\n```\r\nAWS_DEFAULT_PROFILE=default\r\n\r\nexport DOCKER_STORAGE=btrfs\r\nexport KUBERNETES_PROVIDER=aws\r\nexport KUBE_AWS_ZONE=us-west-2a\r\nexport KUBE_ENABLE_CLUSTER_LOGGING=false\r\nexport KUBE_ENABLE_CLUSTER_MONITORING=none\r\nexport MULTIZONE=1\r\nexport NODE_ROOT_DISK_SIZE=32\r\nexport NODE_SIZE=m4.xlarge\r\nexport NUM_NODES=5\r\n```\r\n\r\nWhen the node disk space is low enough to trigger GC, nothing happens.\r\n\r\n```\r\n# df -h\r\nFilesystem      Size  Used Avail Use% Mounted on\r\n/dev/xvda1       32G   28G  3.0G  91% /\r\nudev             10M     0   10M   0% /dev\r\ntmpfs           3.2G  282M  2.9G   9% /run\r\ntmpfs           7.9G  1.1M  7.9G   1% /dev/shm\r\ntmpfs           5.0M     0  5.0M   0% /run/lock\r\ntmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup\r\n\r\n# journalctl -u kubelet | grep -i garbage\r\nApr 20 17:48:20 ip-172-20-0-149 kubelet[4441]: E0420 17:48:20.680505    4441 kubelet.go:956] Image garbage collection failed: unable to find data for container /\r\nMay 19 18:36:30 ip-172-20-0-149 kubelet[27507]: E0519 18:36:30.108168   27507 kubelet.go:956] Image garbage collection failed: unable to find data for container /\r\n```\r\n\r\ncAdvisor output looks OK:\r\n\r\n```\r\n# curl http://127.0.0.1:4194/validate/\r\n...\r\nDocker driver setup: [Supported and recommended]\r\n    Docker exec driver is native-0.2. Storage driver is aufs.\r\n    Docker container state directory is at \"/var/lib/docker/containers\" and is accessible.\r\n\r\n\r\nBlock device setup: [Supported and recommended]\r\n    At least one device supports 'cfq' I/O scheduler. Some disk stats can be reported.\r\n     Disk \"xvda\" Scheduler type \"cfq\".\r\n...\r\n```\r\n\r\nEverything else in the cluster seems to be working.  Any ideas on how to debug?  For now I manually removed dangling and older Docker images."}}
{"actor":{"display_login":"azell"}}