{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37853","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37853/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37853/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37853/events","html_url":"https://github.com/kubernetes/kubernetes/issues/37853","id":192961475,"number":37853,"title":"root cause kernel soft lockups ","user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"labels":[{"id":114528068,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/critical-urgent","name":"priority/critical-urgent","color":"e11d21","default":false},{"id":435667038,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-blocker","name":"release-blocker","color":"d93f0b","default":false}],"state":"open","locked":false,"assignee":{"login":"freehan","id":1332699,"avatar_url":"https://avatars.githubusercontent.com/u/1332699?v=3","gravatar_id":"","url":"https://api.github.com/users/freehan","html_url":"https://github.com/freehan","followers_url":"https://api.github.com/users/freehan/followers","following_url":"https://api.github.com/users/freehan/following{/other_user}","gists_url":"https://api.github.com/users/freehan/gists{/gist_id}","starred_url":"https://api.github.com/users/freehan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freehan/subscriptions","organizations_url":"https://api.github.com/users/freehan/orgs","repos_url":"https://api.github.com/users/freehan/repos","events_url":"https://api.github.com/users/freehan/events{/privacy}","received_events_url":"https://api.github.com/users/freehan/received_events","type":"User","site_admin":false},"assignees":[{"login":"freehan","id":1332699,"avatar_url":"https://avatars.githubusercontent.com/u/1332699?v=3","gravatar_id":"","url":"https://api.github.com/users/freehan","html_url":"https://github.com/freehan","followers_url":"https://api.github.com/users/freehan/followers","following_url":"https://api.github.com/users/freehan/following{/other_user}","gists_url":"https://api.github.com/users/freehan/gists{/gist_id}","starred_url":"https://api.github.com/users/freehan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freehan/subscriptions","organizations_url":"https://api.github.com/users/freehan/orgs","repos_url":"https://api.github.com/users/freehan/repos","events_url":"https://api.github.com/users/freehan/events{/privacy}","received_events_url":"https://api.github.com/users/freehan/received_events","type":"User","site_admin":false},{"login":"gmarek","id":1703842,"avatar_url":"https://avatars.githubusercontent.com/u/1703842?v=3","gravatar_id":"","url":"https://api.github.com/users/gmarek","html_url":"https://github.com/gmarek","followers_url":"https://api.github.com/users/gmarek/followers","following_url":"https://api.github.com/users/gmarek/following{/other_user}","gists_url":"https://api.github.com/users/gmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/gmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmarek/subscriptions","organizations_url":"https://api.github.com/users/gmarek/orgs","repos_url":"https://api.github.com/users/gmarek/repos","events_url":"https://api.github.com/users/gmarek/events{/privacy}","received_events_url":"https://api.github.com/users/gmarek/received_events","type":"User","site_admin":false},{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestone/23","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":124,"closed_issues":995,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-12-08T13:02:44Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":86,"created_at":"2016-12-01T21:02:51Z","updated_at":"2016-12-08T16:19:03Z","closed_at":null,"body":"We're seeing an abnormal number of soft lockups, pegged cpus and unusable nodes recently. The last repros were from @bowei and @freehan. \r\n\r\n**Symptoms**\r\nKernel logs showed ebtables related traces, CPU was pegged at 100% and remained there, no ssh, basically no clarity till the node was reset. Looking back through older test logs, there are several failures that had NotReady nodes that MIGHT be the same bug. \r\n\r\n**Actions**\r\nMinhan's working on a repro and @dchen1107 is trying to figure out when this spike happened by spelunking test logs. \r\n\r\nWe've been syncing iptables rules from kube-proxy more often than we need to for a couple of releases (https://github.com/kubernetes/kubernetes/issues/26637), this is the only thing that springs to mind that might cause cpu spikes. \r\n\r\nWe should probably try to mitigate for 1.5, marking as release-blocker till we have a better handle. \r\n@saad-ali @kubernetes/sig-network @kubernetes/sig-node \r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/265781847","html_url":"https://github.com/kubernetes/kubernetes/issues/37853#issuecomment-265781847","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37853","id":265781847,"user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T16:19:03Z","updated_at":"2016-12-08T16:19:03Z","body":"One of the 1.5+patches machines died overnight.  I am moving all 4 machines to 1.4.6 kubelet and kube-proxy."}}
{"actor":{"display_login":"thockin"}}