{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23762","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23762/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23762/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23762/events","html_url":"https://github.com/kubernetes/kubernetes/issues/23762","id":145282348,"number":23762,"title":"problems with kube-dns pod in juju provider cluster: crashloopbackup error, due to skydns container error","user":{"login":"eddytruyen","id":11139984,"avatar_url":"https://avatars.githubusercontent.com/u/11139984?v=3","gravatar_id":"","url":"https://api.github.com/users/eddytruyen","html_url":"https://github.com/eddytruyen","followers_url":"https://api.github.com/users/eddytruyen/followers","following_url":"https://api.github.com/users/eddytruyen/following{/other_user}","gists_url":"https://api.github.com/users/eddytruyen/gists{/gist_id}","starred_url":"https://api.github.com/users/eddytruyen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eddytruyen/subscriptions","organizations_url":"https://api.github.com/users/eddytruyen/orgs","repos_url":"https://api.github.com/users/eddytruyen/repos","events_url":"https://api.github.com/users/eddytruyen/events{/privacy}","received_events_url":"https://api.github.com/users/eddytruyen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-01T20:50:39Z","updated_at":"2016-04-01T20:50:39Z","closed_at":null,"body":"I executed `juju quickstart u/containers/kubernetes-core/0` on Openstack using an Ubuntu Trusty 14.04 image\r\n\r\nI have encountered an issue with the kube-dns add-on service, in particular the skydns container goes into error.\r\n\r\nBelow you find the relevant info.\r\n\r\n\r\n\r\n```\r\nkubectubuntu@juju-openstack-machine-5:~$ kubectl cluster-info\r\nKubernetes master is running at http://localhost:8080\r\nKubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns\r\n\r\nubuntu@juju-openstack-machine-5:~$ curl http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns\r\n{\r\n\"kind\": \"Status\",\r\n\"apiVersion\": \"v1\",\r\n\"metadata\": {},\r\n\"status\": \"Failure\",\r\n\"message\": \"no endpoints available for service \\\"kube-dns\\\"\",\r\n\"reason\": \"ServiceUnavailable\",\r\n\"code\": 503\r\n}\r\n```\r\n\r\nIt consists of one pod that is in error-modus\r\n\r\n```\r\nubuntu@juju-openstack-machine-5:~$ kubectl get pods --namespace=kube-system\r\nNAME READY STATUS RESTARTS AGE\r\nkube-dns-v8-an2u1 3/4 CrashLoopBackOff 37 2h\r\n```\r\n\r\nIt seems the skydns container is in error mode:\r\n\r\n```\r\nubuntu@juju-openstack-machine-5:~$ kubectl describe pod kube-dns-v8-an2u1 --namespace=kube-system\r\nName: kube-dns-v8-an2u1\r\nNamespace: kube-system\r\nImage(s): gcr.io/google_containers/etcd:2.0.9,gcr.io/google_containers/kube2sky:1.11,gcr.io/google_containers/skydns:2015-03-11-001,gcr.io/google_containers/exechealthz:1.0\r\nNode: 172.17.13.197/172.17.13.197\r\nStart Time: Thu, 31 Mar 2016 13:23:50 +0000\r\nLabels: k8s-app=kube-dns,kubernetes.io/cluster-service=true,version=v8\r\nStatus: Running\r\nReason:\r\nMessage:\r\nIP: 10.1.90.4\r\nReplication Controllers: kube-dns-v8 (1/1 replicas created)\r\nContainers:\r\n<skip etcd:>\r\n<skip kube2sky>\r\nskydns:\r\nContainer ID: docker://96a3b834068ecc30d2083682e9708b30d5cc94c01447f05209d714b7b663e32c\r\nImage: gcr.io/google_containers/skydns:2015-03-11-001\r\nImage ID: docker://sha256:2af62c60205d6bc3da76743366c3cdffe430682d13c3a2249f68e5f7778c23cd\r\nQoS Tier:\r\ncpu: Guaranteed\r\nmemory: Guaranteed\r\nLimits:\r\ncpu: 100m\r\nmemory: 50Mi\r\nRequests:\r\ncpu: 100m\r\nmemory: 50Mi\r\nState: Waiting\r\nReason: CrashLoopBackOff\r\nLast Termination State: Terminated\r\nReason: Error\r\nExit Code: 1\r\nStarted: Thu, 31 Mar 2016 16:05:49 +0000\r\nFinished: Thu, 31 Mar 2016 16:05:49 +0000\r\nReady: False\r\nRestart Count: 37\r\nEnvironment Variables:\r\nEvents:\r\nFirstSeen LastSeen Count From SubobjectPath Reason Message\r\n───────── ──────── ───── ──── ───────────── ────── ───────\r\n58m 58m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id cae965601c36\r\n58m 58m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id cae965601c36\r\n53m 53m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id e090abce9596\r\n53m 53m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id e090abce9596\r\n48m 48m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id ae8879ed723d\r\n48m 48m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id ae8879ed723d\r\n43m 43m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id 1aed82250974\r\n43m 43m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id 1aed82250974\r\n37m 37m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id 48478ef60c17\r\n37m 37m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id 48478ef60c17\r\n32m 32m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id c015f4de7da7\r\n32m 32m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id c015f4de7da7\r\n27m 27m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id af1398492305\r\n27m 27m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id af1398492305\r\n22m 22m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id dbd12d17b612\r\n22m 22m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id dbd12d17b612\r\n17m 17m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id 65c2f150d7ac\r\n17m 17m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id 65c2f150d7ac\r\n12m 12m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id d57988439a4d\r\n12m 12m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id d57988439a4d\r\n6m 6m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id d83992a97bf5\r\n6m 6m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id d83992a97bf5\r\n2h 1m 36 {kubelet 172.17.13.197} spec.containers{skydns} Pulled Container image \"gcr.io/google_containers/skydns:2015-03-11-001\" already present on machine\r\n1m 1m 1 {kubelet 172.17.13.197} spec.containers{skydns} Started Started with docker id 96a3b834068e\r\n1m 1m 1 {kubelet 172.17.13.197} spec.containers{skydns} Created Created with docker id 96a3b834068e\r\n2h 5s 941 {kubelet 172.17.13.197} spec.containers{skydns} Backoff Back-off restarting failed docker container\r\n```\r\n\r\n\r\n\r\nThe logs of skydns show the following message\r\n\r\n```\r\nubuntu@juju-openstack-machine-5:~$ kubectl logs kube-dns-v8-an2u1 --namespace=kube-system -c skydns\r\n\r\nPod \"kube-dns-v8-an2u1\" in namespace \"kube-system\": container \"skydns\" is in waiting state.\r\n```\r\n\r\n\r\nThe yaml description of the skydnscontainer as part of the kube-dnspod:\r\n\r\n```\r\n- args:\r\n- -machines=http://localhost:4001\r\n- -addr=0.0.0.0:53 - -domain=cluster.local.\r\nimage: gcr.io/google_containers/skydns:2015-03-11-001\r\nimagePullPolicy: IfNotPresent\r\nlivenessProbe:\r\nhttpGet:\r\npath: /healthz\r\nport: 8080\r\nscheme: HTTP\r\ninitialDelaySeconds: 30\r\ntimeoutSeconds: 5\r\nname: skydns\r\nports:\r\n- containerPort: 53\r\nname: dns\r\nprotocol: UDP\r\n- containerPort: 53\r\nname: dns-tcp\r\nprotocol: TCP\r\nresources:\r\nlimits:\r\ncpu: 100m\r\nmemory: 50Mi\r\nrequests:\r\ncpu: 100m\r\nmemory: 50Mi\r\nterminationMessagePath: /dev/termination-log\r\n```\r\n\r\nWhen I start a new skydns container via docker on the same node, the container doesn't seem to crash: \r\n\r\n```\r\nubuntu@juju-openstack-machine-6:~$ docker run -it gcr.io/google_containers/skydns:2015-03-11-001\r\n2016/03/31 16:28:46 skydns: falling back to default configuration, could not read from etcd: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2016/03/31 16:28:46 skydns: ready for queries on skydns.local. for tcp://127.0.0.1:53 [rcache 0]\r\n2016/03/31 16:28:46 skydns: ready for queries on skydns.local. for udp://127.0.0.1:53 [rcache 0]\r\n```"}}