{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22042","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22042/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22042/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22042/events","html_url":"https://github.com/kubernetes/kubernetes/issues/22042","id":136607947,"number":22042,"title":"skydns not working properly","user":{"login":"mfanjie","id":16643108,"avatar_url":"https://avatars.githubusercontent.com/u/16643108?v=3","gravatar_id":"","url":"https://api.github.com/users/mfanjie","html_url":"https://github.com/mfanjie","followers_url":"https://api.github.com/users/mfanjie/followers","following_url":"https://api.github.com/users/mfanjie/following{/other_user}","gists_url":"https://api.github.com/users/mfanjie/gists{/gist_id}","starred_url":"https://api.github.com/users/mfanjie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfanjie/subscriptions","organizations_url":"https://api.github.com/users/mfanjie/orgs","repos_url":"https://api.github.com/users/mfanjie/repos","events_url":"https://api.github.com/users/mfanjie/events{/privacy}","received_events_url":"https://api.github.com/users/mfanjie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-02-26T06:16:03Z","updated_at":"2016-02-26T09:59:12Z","closed_at":null,"body":"We are using K8S as our private container engine and we ran into several problems on skydns addons, our host OS is fedora\r\n[fedora@kubernetes-minion1 ~]$ cat /etc/issue\r\nFedora release 21 (Twenty One)\r\nKernel \\r on an \\m (\\l)\r\n####Problem description\r\n1). kube2sky terminated with OOMKilled error\r\n PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND\r\n    1     0 root     S     392m  0.3   5  0.0 /kube2sky -domain=kubernetes.local\r\nWe fixed this problem by remove the resource limit on kube2sky container, and I found previous discussion on the memory setting for those containers in issue #8042, seems in our environment, the memory consume of kube2sky is extremely high.\r\nOur env info is like following\r\nkubectl get pods --all-namespaces|wc -l\r\n     310\r\nkubectl get svc --all-namespaces|wc -l\r\n     117\r\nkubectl get nodes|wc -l\r\n      54\r\n2) skydns container got killed repeatedly\r\ncheck pod description\r\nskydns:\r\n    Container ID:\tdocker://d4c3f3273d34443bac0da954ebfd3319ad53681a6ac3964f89dbcecda401b7c4\r\n    Image:\t\tconatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c\r\n    Image ID:\t\tdocker://a0835f7bffb9231347e5bbcb5a68f0ed45598978a9b15365ba440a136a5097d6\r\n    QoS Tier:\r\n      cpu:\tGuaranteed\r\n      memory:\tGuaranteed\r\n    Limits:\r\n      memory:\t50Mi\r\n      cpu:\t100m\r\n    Requests:\r\n      cpu:\t\t\t100m\r\n      memory:\t\t\t50Mi\r\n    State:\t\t\tRunning\r\n      Started:\t\t\tFri, 26 Feb 2016 10:27:50 +0800\r\n    Last Termination State:\tTerminated\r\n      Reason:\t\t\tError\r\n      Exit Code:\t\t2\r\n      Started:\t\t\tFri, 26 Feb 2016 10:27:10 +0800\r\n      Finished:\t\t\tFri, 26 Feb 2016 10:27:48 +0800\r\n    Ready:\t\t\tTrue\r\n    Restart Count:\t\t27\r\n    Environment Variables:\r\n...\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\t\t\t\t\t\tSubobjectPath\t\tReason\tMessage\r\n  ─────────\t────────\t─────\t────\t\t\t\t\t\t\t\t─────────────\t\t──────\t───────\r\n 1m\t1m\t1\t{kubelet kubernetes-minion1}\tspec.containers{skydns}\tCreated\t\tCreated with docker id e5712f554649\r\n  1m\t1m\t1\t{kubelet kubernetes-minion1}\tspec.containers{skydns}\tStarted\t\tStarted with docker id e5712f554649\r\n  19h\t18s\t1462\t{kubelet kubernetes-minion1}\tspec.containers{skydns}\tUnhealthy\tReadiness probe failed: Get http://172.20.124.131:8080/healthz: dial tcp 172.20.124.131:8080: network is unreachable\r\n  19h\t18s\t1235\t{kubelet kubernetes-minion1}\tspec.containers{skydns}\tUnhealthy\tLiveness probe failed: Get http://172.20.124.131:8080/healthz: dial tcp 172.20.124.131:8080: network is unreachable\r\n  17s\t17s\t1\t{kubelet kubernetes-minion1}\tspec.containers{skydns}\tKilling\t\tKilling with docker id e5712f554649\r\n\r\nI put script on both healthz and skydns to wget http://172.20.124.131:8080 and http://127.0.0.1:8080, and both works, the only different in very short time(seconds to minutes), container skydns is killed with above error.\r\n\r\nthe following is logs on the minion, please note nslookup works when skydns container is running.\r\n\r\nFeb 26 02:31:33 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:33Z\" level=info msg=\"GET /images/conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c/json\"\r\nFeb 26 02:31:33 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:33Z\" level=info msg=\"+job image_inspect(conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c)\"\r\nFeb 26 02:31:33 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:33Z\" level=info msg=\"-job image_inspect(conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c) = OK (0)\"\r\nFeb 26 02:31:33 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:33Z\" level=info msg=\"POST /containers/create?name=k8s_skydns.a2cb16fd_kube-dns-v10-etqab_kube-system_7636e496-db7d-11e5-8ed1-74dbd1a04bc0_c5e1ee82\"\r\nFeb 26 02:31:33 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:33Z\" level=info msg=\"+job create(k8s_skydns.a2cb16fd_kube-dns-v10-etqab_kube-system_7636e496-db7d-11e5-8ed1-74dbd1a04bc0_c5e1ee82)\"\r\nFeb 26 02:31:34 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:34Z\" level=info msg=\"+job log(create, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c)\"\r\nFeb 26 02:31:34 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:34Z\" level=info msg=\"-job log(create, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c) = OK (0)\"\r\nFeb 26 02:31:34 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:34Z\" level=info msg=\"-job create(k8s_skydns.a2cb16fd_kube-dns-v10-etqab_kube-system_7636e496-db7d-11e5-8ed1-74dbd1a04bc0_c5e1ee82) = OK (0)\"\r\nFeb 26 02:31:34 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:34Z\" level=info msg=\"+job log(start, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c)\"\r\nFeb 26 02:31:34 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:34Z\" level=info msg=\"-job log(start, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c) = OK (0)\"\r\nFeb 26 02:31:35 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:31:35Z\" level=error msg=\"Unable to RegisterMachine k8s_skydns.a2cb16fd_kube-dns-v10-etqab_kube-system_7636e496-db7d-11e5-8ed1-74dbd1a04bc0_c5e1ee82 for 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f: Invalid machine name\"\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 kubelet[18925]: I0226 02:32:07.056562   18925 prober.go:129] Liveness probe for \"kube-dns-v10-etqab_kube-system:skydns\" failed (failure): Get http://172.20.124.131:8080/healthz: dial tcp 172.20.124.131:8080: network is unreachable\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 kubelet[18925]: I0226 02:32:07.056648   18925 manager.go:1769] pod \"kube-dns-v10-etqab_kube-system\" container \"skydns\" is unhealthy (probe result: failure), it will be killed and re-created.\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 kubelet[18925]: I0226 02:32:07.057023   18925 manager.go:1404] Killing container \"12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f skydns kube-system/kube-dns-v10-etqab\" with 30 second grace period\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:32:07Z\" level=info msg=\"+job log(die, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c)\"\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:32:07Z\" level=info msg=\"-job log(die, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c) = OK (0)\"\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:32:07Z\" level=info msg=\"+job log(stop, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c)\"\r\nFeb 26 02:32:07 kubernetes-minion-52-3770 docker[11798]: time=\"2016-02-26T02:32:07Z\" level=info msg=\"-job log(stop, 12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f, conatiner.repositry/kubernetes/skydns:2015-10-13-8c72f8c) = OK (0)\"\r\nFeb 26 02:32:08 kubernetes-minion-52-3770 kubelet[18925]: I0226 02:32:08.177747   18925 manager.go:1438] Container \"12e11c1fceac2ee102a69ff79faf479ef23a5be10bdc43aa432275dd345dc21f skydns kube-system/kube-dns-v10-etqab\" exited after 1.120643896s"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/189200680","html_url":"https://github.com/kubernetes/kubernetes/issues/22042#issuecomment-189200680","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22042","id":189200680,"user":{"login":"mfanjie","id":16643108,"avatar_url":"https://avatars.githubusercontent.com/u/16643108?v=3","gravatar_id":"","url":"https://api.github.com/users/mfanjie","html_url":"https://github.com/mfanjie","followers_url":"https://api.github.com/users/mfanjie/followers","following_url":"https://api.github.com/users/mfanjie/following{/other_user}","gists_url":"https://api.github.com/users/mfanjie/gists{/gist_id}","starred_url":"https://api.github.com/users/mfanjie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfanjie/subscriptions","organizations_url":"https://api.github.com/users/mfanjie/orgs","repos_url":"https://api.github.com/users/mfanjie/repos","events_url":"https://api.github.com/users/mfanjie/events{/privacy}","received_events_url":"https://api.github.com/users/mfanjie/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T09:59:12Z","updated_at":"2016-02-26T09:59:12Z","body":"one quick update, after I recompile skydns and exechealthz with latest code from community and rebuild image. The problem was gone, keep monitoring over weekend."}}