{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubeadm/issues/684","repository_url":"https://api.github.com/repos/kubernetes/kubeadm","labels_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/684/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/684/comments","events_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/684/events","html_url":"https://github.com/kubernetes/kubeadm/issues/684","id":293050479,"number":684,"title":"Kubeadm init blocks at \"This might take a minute or longer if the control plane images have to be pulled","user":{"login":"PhoenixAD","id":21970244,"avatar_url":"https://avatars3.githubusercontent.com/u/21970244?v=4","gravatar_id":"","url":"https://api.github.com/users/PhoenixAD","html_url":"https://github.com/PhoenixAD","followers_url":"https://api.github.com/users/PhoenixAD/followers","following_url":"https://api.github.com/users/PhoenixAD/following{/other_user}","gists_url":"https://api.github.com/users/PhoenixAD/gists{/gist_id}","starred_url":"https://api.github.com/users/PhoenixAD/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PhoenixAD/subscriptions","organizations_url":"https://api.github.com/users/PhoenixAD/orgs","repos_url":"https://api.github.com/users/PhoenixAD/repos","events_url":"https://api.github.com/users/PhoenixAD/events{/privacy}","received_events_url":"https://api.github.com/users/PhoenixAD/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-01-31T06:58:06Z","updated_at":"2018-02-28T11:13:04Z","closed_at":null,"author_association":"NONE","body":"## Versions\r\n\r\n**kubeadm version** (use `kubeadm version`):\r\n\r\n**Environment**:\r\n- **Kubernetes version** (use `kubectl version`):v1.9.2\r\n- **Cloud provider or hardware configuration**:Virtual Box \r\n- **OS** (e.g. from /etc/os-release):Ubuntu 16.04.0 LTS (Xeniak Xerus) amd64\r\n- **Kernel** (e.g. `uname -a`):linux 4.4.0-62-generic\r\n- **Others**:kubeadm version :v1.9.2:  amd64,  kubelet version :v1.9.2 amd64, kubernetes-cni version :0.6.0-00 amd64 ,docker version:17.03.2-ce \r\n\r\n\r\n## What happened?\r\nAs I try to run kubeadm init,it hangs with \r\n**xx@xx:~$ sudo kubeadm init  --kubernetes-version=v1.9.2**\r\n\r\n> \r\n[init] Using Kubernetes version: v1.9.2 \r\n[init] Using Authorization modes: [Node RBAC]\r\n[preflight] Running pre-flight checks.\r\n\t[WARNING FileExisting-crictl]: crictl not found in system path\r\n[preflight] Starting the kubelet service\r\n[certificates] Generated ca certificate and key.\r\n[certificates] Generated apiserver certificate and key.\r\n[certificates] apiserver serving cert is signed for DNS names [kickseed kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.17.41.15]\r\n[certificates] Generated apiserver-kubelet-client certificate and key.\r\n[certificates] Generated sa key and public key.\r\n[certificates] Generated front-proxy-ca certificate and key.\r\n[certificates] Generated front-proxy-client certificate and key.\r\n[certificates] Valid certificates and keys now exist in \"/etc/kubernetes/pki\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"admin.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"kubelet.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"controller-manager.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"scheduler.conf\"\r\n[controlplane] Wrote Static Pod manifest for component kube-apiserver to \"/etc/kubernetes/manifests/kube-apiserver.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-controller-manager to \"/etc/kubernetes/manifests/kube-controller-manager.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-scheduler to \"/etc/kubernetes/manifests/kube-scheduler.yaml\"\r\n[etcd] Wrote Static Pod manifest for a local etcd instance to \"/etc/kubernetes/manifests/etcd.yaml\"\r\n[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory \"/etc/kubernetes/manifests\".\r\n[init] This might take a minute or longer if the control plane images have to be pulled.\r\n\r\n\r\nThen I check the kubelet log:\r\n**xx@xx:~$ sudo journalctl -xeu  kubelet:**\r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.280984 28516 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281317 28516 kuberuntime_sandbox.go:54] CreatePodSandbox for pod \"kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281580 28516 kuberuntime_manager.go:647] createPodSandbox for pod \"kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281875 28516 pod_workers.go:186] Error syncing pod 69c12074e336b0dbbd0a1666ce05226a (\"kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)\"), skipping: failed to \"CreatePodSandbox\" for \"kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)\\\" failed: rpc error: code = Unknown desc = failed pulling image \\\"gcr.io/google_containers/pause-amd64:3.0\\\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout\" \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.380290 28516 event.go:209] Unable to write event: 'Patch https://172.17.41.15:6443/api/v1/namespaces/default/events/kickseed.150ecf46afb098b7: dial tcp 172.17.41.15:6443: getsockopt: connection refused' (may retry after sleeping) \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.933783 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.934707 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.935921 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281024 28516 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281352 28516 kuberuntime_sandbox.go:54] CreatePodSandbox for pod \"kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281634 28516 kuberuntime_manager.go:647] createPodSandbox for pod \"kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281938 28516 pod_workers.go:186] Error syncing pod 6546d6faf0b50c9fc6712ce25ee9b6cb (\"kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)\"), skipping: failed to \"CreatePodSandbox\" for \"kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)\\\" failed: rpc error: code = Unknown desc = failed pulling image \\\"gcr.io/google_containers/pause-amd64:3.0\\\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout\" \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.934694 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.935613 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.936669 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: W0131 14:45:05.073692 28516 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d \r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.074106 28516 kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized\r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.935680 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.937423 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.937963 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:45:05 kickseed kubelet[28516]: I0131 14:45:05.974034 28516 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach \r\n**Jan** 31 14:45:06 kickseed kubelet[28516]: I0131 14:45:06.802447 28516 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach \r\n**Jan** 31 14:45:06 kickseed kubelet[28516]: I0131 14:45:06.804242 28516 kubelet_node_status.go:82] Attempting to register node kickseed\r\n**Jan** 31 14:45:06 kickseed kubelet[28516]: E0131 14:45:06.804778 28516 kubelet_node_status.go:106] Unable to register node \"kickseed\" with API server: Post https://172.17.41.15:6443/api/v1/nodes: dial tcp 172.17.41.15:6443: getsockopt: con\r\n   \r\n\r\n\r\n**xx@xx:~$ sudo systemctl status kubelet:**\r\n\r\nkubelet.service - kubelet: The Kubernetes Node Agent \r\nLoaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) \r\nDrop-In: /etc/systemd/system/kubelet.service.d \r\n        └─11-kubeadm.conf, 10-kubeadm1.conf, 90-local-extras.conf \r\nActive: active (running) since Wed 2018-01-31 13:53:46 CST; 49min ago \r\n     Docs: http://kubernetes.io/docs/ \r\nMain PID: 28516 (kubelet) \r\n     Tasks: 13 \r\nMemory: 37.8M \r\n     CPU: 22.767s \r\nCGroup: /system.slice/kubelet.service\r\n            └─28516 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cgroup-driver=cgroupfs --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --fail-swap-on=false \r\n\r\n**Jan** 31 14:43:17 kickseed kubelet[28516]: E0131 14:43:17.862590 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:17 kickseed kubelet[28516]: E0131 14:43:17.863474 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.621818 28516 event.go:209] Unable to write event: 'Patch https://172.17.41.15:6443/api/v1/namespaces/default/events/kickseed.150ecf46afb098b7: dial tcp 172.17.41.15:6443: getsockopt: connection refused' (may retry after sleeping) \r\n**Jan** 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.862440 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.863379 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.864424 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.255460 28516 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node \"kickseed\" not found \r\n**Jan** 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.863266 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.864238 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused \r\n**Jan** 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.865262 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused\r\n\r\n\r\n**Some docker images are listed as followed:**\r\ngcr.io/google_containers/kube-apiserver-amd64:v1.9.2\r\ngcr.io/google_containers/kube-controller-manager-amd64:v1.9.2\r\ngcr.io/google_containers/kube-scheduler-amd64:v1.9.2 \r\ngcr.io/google_containers/kube-proxy-amd64:v1.9.2 \r\ngcr.io/google_containers/etcd-amd64:3.2.14 \r\ngcr.io/google_containers/pause-amd64:3.1\r\ngcr.io/google_containers/kube-dnsmasq-amd64:1.4.1\r\ngcr.io/google_containers/kubernetes-dashboard-amd64:v1.8.2 \r\ngcr.io/google_containers/kubedns-amd64:1.9\r\ngcr.io/google_containers/kube-discovery-amd64:1.0\r\ngcr.io/google_containers/exechealthz-amd64:v1.2.0 \r\ngcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8\r\ngcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8\r\ngcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8\r\ngcr.io/google_containers/dnsmasq-metrics-amd64:1.0.1\r\n\r\n\r\n\r\n## What you expected to happen?\r\nkubeadm init should complete\r\n\r\n## How to reproduce it (as minimally and precisely as possible)?\r\nvirtualbox with Ubuntu 16.04 and kubeadm 1.9.2\r\n\r\n## Anything else we need to know?\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubeadm/issues/comments/369207510","html_url":"https://github.com/kubernetes/kubeadm/issues/684#issuecomment-369207510","issue_url":"https://api.github.com/repos/kubernetes/kubeadm/issues/684","id":369207510,"user":{"login":"simis2626","id":18462964,"avatar_url":"https://avatars1.githubusercontent.com/u/18462964?v=4","gravatar_id":"","url":"https://api.github.com/users/simis2626","html_url":"https://github.com/simis2626","followers_url":"https://api.github.com/users/simis2626/followers","following_url":"https://api.github.com/users/simis2626/following{/other_user}","gists_url":"https://api.github.com/users/simis2626/gists{/gist_id}","starred_url":"https://api.github.com/users/simis2626/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simis2626/subscriptions","organizations_url":"https://api.github.com/users/simis2626/orgs","repos_url":"https://api.github.com/users/simis2626/repos","events_url":"https://api.github.com/users/simis2626/events{/privacy}","received_events_url":"https://api.github.com/users/simis2626/received_events","type":"User","site_admin":false},"created_at":"2018-02-28T11:13:04Z","updated_at":"2018-02-28T11:13:04Z","author_association":"NONE","body":"+1"}}
{"actor":{"display_login":"simis2626"}}