{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28712","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28712/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28712/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28712/events","html_url":"https://github.com/kubernetes/kubernetes/issues/28712","id":164623629,"number":28712,"title":"GCE IPv4: martian source errors and Node becomes inoperable","user":{"login":"chrislovecnm","id":335402,"avatar_url":"https://avatars.githubusercontent.com/u/335402?v=3","gravatar_id":"","url":"https://api.github.com/users/chrislovecnm","html_url":"https://github.com/chrislovecnm","followers_url":"https://api.github.com/users/chrislovecnm/followers","following_url":"https://api.github.com/users/chrislovecnm/following{/other_user}","gists_url":"https://api.github.com/users/chrislovecnm/gists{/gist_id}","starred_url":"https://api.github.com/users/chrislovecnm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrislovecnm/subscriptions","organizations_url":"https://api.github.com/users/chrislovecnm/orgs","repos_url":"https://api.github.com/users/chrislovecnm/repos","events_url":"https://api.github.com/users/chrislovecnm/events{/privacy}","received_events_url":"https://api.github.com/users/chrislovecnm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-07-08T21:53:46Z","updated_at":"2016-07-08T22:49:50Z","closed_at":null,"body":"Hi all\r\n\r\nI am running 1.3 beta, and have had 2 nodes that started in a state where the kernel was throwing martian source errors, and two nodes went to that state.\r\n\r\nHere is a snippet of the kernel logs:\r\n\r\n```\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.359924] Memory cgroup stats for /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426: cache:100KB rss:204700KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:204792KB inactive_file:8KB active_file:0KB unevictable:0KB\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.402989] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.413053] [ 4051]     0  4051     1112      190       6        0           994 sh\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.429117] [ 4057]     0  4057    27225     6949      55        0           994 google-fluentd\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.441524] [ 4062]     0  4062   274400    48880     182        0           994 google-fluentd\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.451015] Memory cgroup out of memory: Kill process 4062 (google-fluentd) score 1948 or sacrifice child\r\nJul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.463970] Killed process 4062 (google-fluentd) total-vm:1097600kB, anon-rss:187684kB, file-rss:7836kB\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.207417] google-fluentd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=994\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.216157] google-fluentd cpuset=da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426 mems_allowed=0\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.227357] CPU: 5 PID: 28307 Comm: google-fluentd Tainted: G         C    3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707] Hardware name: Google Google, BIOS Google 01/01/2011\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  0000000000000000 ffffffff8150e835 ffff880798508960 ffff8807978efc00\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  ffffffff8150c431 ffff880797f1f060 ffffffff810a7807 0000000000000001\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  ffff880797f1f040 ffff880798b01500 0000000000000001 0000000000000000\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707] Call Trace:\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8150e835>] ? dump_stack+0x5d/0x78\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8150c431>] ? dump_header+0x76/0x1e8\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff810a7807>] ? __wake_up_common+0x57/0x90\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8114073d>] ? find_lock_task_mm+0x3d/0x90\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81140b7d>] ? oom_kill_process+0x21d/0x370\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8114073d>] ? find_lock_task_mm+0x3d/0x90\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff811a087a>] ? mem_cgroup_oom_synchronize+0x52a/0x590\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8119fe00>] ? mem_cgroup_try_charge_mm+0xa0/0xa0\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81141330>] ? pagefault_out_of_memory+0x10/0x80\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81057505>] ? __do_page_fault+0x3c5/0x4f0\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81171b5f>] ? mprotect_fixup+0x14f/0x270\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81516a28>] ? page_fault+0x28/0x30\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.356846] Task in /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426 killed as a result of limit of /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.381018] memory: usage 204800kB, limit 204800kB, failcnt 231\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.387462] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.396160] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.404259] Memory cgroup stats for /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426: cache:100KB rss:204700KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:204792KB inactive_file:8KB active_file:0KB unevictable:0KB\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.438068] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.446401] [ 4051j     0  4051     1112      190       6        0           994 sh\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.455984] [ 4057]     0  4057    27225     6949      55        0           994 google-fluentd\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.466727] [28303]     0 28303   289117    48527     171        0           994 google-fluentd\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.477300] Memory cgroup out of memory: Kill process 28303 (google-fluentd) score 1941 or sacrifice child\r\nJul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.488688] Killed process 28303 (google-fluentd) total-vm:1156468kB, anon-rss:186272kB, file-rss:7836kB\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.199744] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.207097] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.216441] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.225091] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.244086] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.252793] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.448076] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0\r\nJul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.455732] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...\r\n```\r\nThe cluster is over 1k in nodes, and is running 1k in Pet Set Cassandra Pets.\r\n\r\nLinux info\r\n\r\n```console\r\n$ uname -a\r\nLinux kubernetes-minion-group-n0xa 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux\r\n```\r\n\r\nDebian version\r\n\r\n```console\r\n$ cat /etc/debian_version\r\n7.11\r\n```\r\n\r\nWhat other info is required?\r\n\r\ncc: @bgrant0607 @thockin @Random-Liu "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/231490956","html_url":"https://github.com/kubernetes/kubernetes/issues/28712#issuecomment-231490956","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/28712","id":231490956,"user":{"login":"chrislovecnm","id":335402,"avatar_url":"https://avatars.githubusercontent.com/u/335402?v=3","gravatar_id":"","url":"https://api.github.com/users/chrislovecnm","html_url":"https://github.com/chrislovecnm","followers_url":"https://api.github.com/users/chrislovecnm/followers","following_url":"https://api.github.com/users/chrislovecnm/following{/other_user}","gists_url":"https://api.github.com/users/chrislovecnm/gists{/gist_id}","starred_url":"https://api.github.com/users/chrislovecnm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrislovecnm/subscriptions","organizations_url":"https://api.github.com/users/chrislovecnm/orgs","repos_url":"https://api.github.com/users/chrislovecnm/repos","events_url":"https://api.github.com/users/chrislovecnm/events{/privacy}","received_events_url":"https://api.github.com/users/chrislovecnm/received_events","type":"User","site_admin":false},"created_at":"2016-07-08T22:49:50Z","updated_at":"2016-07-08T22:49:50Z","body":"Forgot to mention restart of the node fixes this problem."}}
{"actor":{"display_login":"chrislovecnm"}}