{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29645","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29645/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29645/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29645/events","html_url":"https://github.com/kubernetes/kubernetes/issues/29645","id":167731703,"number":29645,"title":"nodes can not auto recover from OutOfDisk. ","user":{"login":"minjs","id":9019394,"avatar_url":"https://avatars.githubusercontent.com/u/9019394?v=3","gravatar_id":"","url":"https://api.github.com/users/minjs","html_url":"https://github.com/minjs","followers_url":"https://api.github.com/users/minjs/followers","following_url":"https://api.github.com/users/minjs/following{/other_user}","gists_url":"https://api.github.com/users/minjs/gists{/gist_id}","starred_url":"https://api.github.com/users/minjs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/minjs/subscriptions","organizations_url":"https://api.github.com/users/minjs/orgs","repos_url":"https://api.github.com/users/minjs/repos","events_url":"https://api.github.com/users/minjs/events{/privacy}","received_events_url":"https://api.github.com/users/minjs/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/support","name":"kind/support","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-07-26T22:57:34Z","updated_at":"2016-07-29T00:20:42Z","closed_at":null,"body":"3 of my 5 nodes hit the OutOfDisk in 2 months ago. And then, those 3 nodes can not be used even the disk usage is dropped. I have to remove the node from cluster and restart the kubelet on that disk. \r\n\r\n\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2016-07-26T22:50:53Z\",\"lastTransitionTime\":\"2016-06-14T21:25:15Z\",\"reason\":\"KubeletReady\",\"message\":\"kubelet is posting ready status\"},{\"type\":\"OutOfDisk\",\"status\":\"Unknown\",\"lastHeartbeatTime\":\"2016-05-16T20:52:46Z\",\"lastTransitionTime\":\"2016-06-14T21:24:49Z\",\"reason\":\"NodeStatusNeverUpdated\",\"message\":\"Kubelet never posted node status.\"}]\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/236062718","html_url":"https://github.com/kubernetes/kubernetes/issues/29645#issuecomment-236062718","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29645","id":236062718,"user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T00:20:42Z","updated_at":"2016-07-29T00:20:42Z","body":"@minjs can you please figure out what's eating your disk? \r\nYou can check if it's container logs by running du within /var/lib/docker/containers/*/*-json.log\r\n\r\n"}}
{"actor":{"display_login":"bprashanth"}}