{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/72","repository_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin","labels_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/72/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/72/comments","events_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/72/events","html_url":"https://github.com/NVIDIA/k8s-device-plugin/issues/72","id":359127481,"node_id":"MDU6SXNzdWUzNTkxMjc0ODE=","number":72,"title":"How to resolve kubelet error after installing nvidia device driver plugin in k8s worker nodes, getting nvidia-smi: dial tcp 172.28.3.100:10250: getsockopt: no route to host","user":{"login":"Bharathkumarraju","id":5610588,"node_id":"MDQ6VXNlcjU2MTA1ODg=","avatar_url":"https://avatars2.githubusercontent.com/u/5610588?v=4","gravatar_id":"","url":"https://api.github.com/users/Bharathkumarraju","html_url":"https://github.com/Bharathkumarraju","followers_url":"https://api.github.com/users/Bharathkumarraju/followers","following_url":"https://api.github.com/users/Bharathkumarraju/following{/other_user}","gists_url":"https://api.github.com/users/Bharathkumarraju/gists{/gist_id}","starred_url":"https://api.github.com/users/Bharathkumarraju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bharathkumarraju/subscriptions","organizations_url":"https://api.github.com/users/Bharathkumarraju/orgs","repos_url":"https://api.github.com/users/Bharathkumarraju/repos","events_url":"https://api.github.com/users/Bharathkumarraju/events{/privacy}","received_events_url":"https://api.github.com/users/Bharathkumarraju/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-09-11T16:35:58Z","updated_at":"2018-09-12T13:38:23Z","closed_at":"2018-09-12T13:38:23Z","author_association":"NONE","body":"I have tried to run nvidia device plugin in k8s gpu nodes and got below errors.\r\n\r\n```\r\n~ kubectl logs nvidia-smi -n dev\r\nError from server: Get https://172.28.3.100:10250/containerLogs/dev/nvidia-smi/nvidia-smi: dial tcp 172.28.3.100:10250: getsockopt: no route to host\r\n```\r\n\r\n```\r\nroot      5413     1  2 11:53 ?        00:04:26 /usr/bin/kubelet --address=0.0.0.0 --authentication-token-webhook --authorization-mode=Webhook --allow-privileged=true --cloud-provider=aws --cluster-domain=cluster.local --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --container-runtime=docker --network-plugin=cni --cgroup-driver=cgroupfs --register-node=true --kubeconfig=/var/lib/kubelet/kubeconfig --feature-gates=RotateKubeletServerCertificate=true --anonymous-auth=false --client-ca-file=/etc/kubernetes/pki/ca.crt\r\nroot      6070  6053  0 11:53 ?        00:00:00 /bin/sh -c kube-proxy --resource-container=\"\" --oom-score-adj=-998 --master=https://adc7d5d2f1c7f338847179ff03e8c45b.sk1.us-east-1.eks.amazonaws.com --kubeconfig=/var/lib/kube-proxy/kubeconfig --proxy-mode=iptables --v=2 1>>/var/log/kube-proxy.log 2>&1\r\nroot      6101  6070  0 11:53 ?        00:00:28 kube-proxy --resource-container= --oom-score-adj=-998 --master=https://adc7d5d2f1c7f338847179ff03e8c45b.sk1.us-east-1.eks.amazonaws.com --kubeconfig=/var/lib/kube-proxy/kubeconfig --proxy-mode=iptables --v=2\r\nnfsnobo+  6352  6330  0 11:53 ?        00:00:01 /kube-rbac-proxy --secure-listen-address=:9100 --upstream=http://127.0.0.1:9101/\r\nroot     32424 32411  0 15:01 pts/0    00:00:00 grep --color=auto -i kube\r\n\r\n```\r\n\r\n\r\n```\r\n[Unit]\r\nDescription=Kubernetes Kubelet\r\nDocumentation=https://github.com/kubernetes/kubernetes\r\nAfter=docker.service\r\nRequires=docker.service\r\n\r\n[Service]\r\nExecStart=/usr/bin/kubelet \\\r\n  --address=0.0.0.0 \\\r\n  --authentication-token-webhook \\\r\n  --authorization-mode=Webhook \\\r\n  --allow-privileged=true \\\r\n  --cloud-provider=aws \\\r\n  --cluster-domain=cluster.local \\\r\n  --cni-bin-dir=/opt/cni/bin \\\r\n  --cni-conf-dir=/etc/cni/net.d \\\r\n  --container-runtime=docker \\\r\n  --network-plugin=cni \\\r\n  --cgroup-driver=cgroupfs \\\r\n  --register-node=true \\\r\n  --kubeconfig=/var/lib/kubelet/kubeconfig \\\r\n  --feature-gates=RotateKubeletServerCertificate=true \\\r\n  --anonymous-auth=false \\\r\n  --client-ca-file=/etc/kubernetes/pki/ca.crt $KUBELET_ARGS $KUBELET_20 $KUBELET_EXTRA_ARGS\r\n\r\nRestart=on-failure\r\nRestartSec=5\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n\r\n```\r\n\r\n\r\n\r\n```\r\n journalctl -u kubelet | grep -i Error\r\nSep 11 11:53:27 ip-172-28-4-241.ec2.internal kubelet[5413]: I0911 11:53:27.254108    5413 docker_service.go:249] Docker Info: &{ID:CA47:CJ5Y:7I2C:VLGO:VW6F:M42Y:DV7R:3RC2:CGSB:BSFJ:DNF7:UXFR Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:17 OomKillDisable:true NGoroutines:26 SystemTime:2018-09-11T11:53:27.246986039Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.14.62-70.117.amzn2.x86_64 OperatingSystem:Amazon Linux 2 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc4201bc310 NCPU:4 MemTotal:64383959040 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-28-4-241.ec2.internal Labels:[] ExperimentalBuild:false ServerVersion:17.06.2-ce ClusterStore: ClusterAdvertise: Runtimes:map[nvidia:{Path:/usr/bin/nvidia-container-runtime Args:[]} runc:{Path:docker-runc Args:[]}] DefaultRuntime:nvidia Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:6e23458c129b551d5c9871e5174f6b1b7f6d1170 Expected:6e23458c129b551d5c9871e5174f6b1b7f6d1170} RuncCommit:{ID:810190ceaa507aa2727d7ae6f4790c76ec150bd2 Expected:810190ceaa507aa2727d7ae6f4790c76ec150bd2} InitCommit:{ID:949e6fa Expected:949e6fa} SecurityOptions:[name=seccomp,profile=default]}\r\nSep 11 11:53:30 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:30.453225    5413 pod_workers.go:186] Error syncing pod 4c7c8c9d-b5b9-11e8-8a8a-123840b2572c (\"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\"), skipping: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]\r\nSep 11 11:53:42 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:42.343110    5413 pod_workers.go:186] Error syncing pod 4c7c8c9d-b5b9-11e8-8a8a-123840b2572c (\"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\"), skipping: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.218733    5413 cni.go:259] Error adding network: add cmd: failed to assign an IP address to container\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.218767    5413 cni.go:227] Error while adding to cni network: add cmd: failed to assign an IP address to container\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.357423    5413 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"nvidia-device-plugin-daemonset-bk8t7_kube-system\" network: add cmd: failed to assign an IP address to container\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.357517    5413 kuberuntime_sandbox.go:54] CreatePodSandbox for pod \"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"nvidia-device-plugin-daemonset-bk8t7_kube-system\" network: add cmd: failed to assign an IP address to container\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.357541    5413 kuberuntime_manager.go:646] createPodSandbox for pod \"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"nvidia-device-plugin-daemonset-bk8t7_kube-system\" network: add cmd: failed to assign an IP address to container\r\nSep 11 11:53:56 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 11:53:56.357659    5413 pod_workers.go:186] Error syncing pod 4c7c8c9d-b5b9-11e8-8a8a-123840b2572c (\"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\"), skipping: failed to \"CreatePodSandbox\" for \"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"nvidia-device-plugin-daemonset-bk8t7_kube-system(4c7c8c9d-b5b9-11e8-8a8a-123840b2572c)\\\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \\\"nvidia-device-plugin-daemonset-bk8t7_kube-system\\\" network: add cmd: failed to assign an IP address to container\"\r\nSep 11 12:05:34 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 12:05:34.617117    5413 endpoint.go:132] listAndWatch ended unexpectedly for device plugin nvidia.com/gpu with error rpc error: code = Unavailable desc = transport is closing\r\nSep 11 12:14:31 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 12:14:31.694845    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 12:35:34 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 12:35:34.162921    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 12:56:36 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 12:56:36.586987    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 13:17:39 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 13:17:39.019506    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 13:38:41 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 13:38:41.502727    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 13:59:43 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 13:59:43.963967    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 14:20:46 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 14:20:46.506590    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 14:41:49 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 14:41:49.027261    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 15:02:51 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 15:02:51.486128    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\nSep 11 15:23:53 ip-172-28-4-241.ec2.internal kubelet[5413]: E0911 15:23:53.959861    5413 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"ip-172-28-4-241.ec2.internal\": Unauthorized\r\n```\r\n\r\n\r\n```\r\n kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.2\", GitCommit:\"81753b10df112992bf51bbc2c2f85208aad78335\", GitTreeState:\"clean\", BuildDate:\"2018-05-12T04:12:47Z\", GoVersion:\"go1.9.6\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.3\", GitCommit:\"2bba0127d85d5a46ab4b778548be28623b32d0b0\", GitTreeState:\"clean\", BuildDate:\"2018-05-28T20:13:43Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n```\r\n\r\n\r\n```\r\ncat /etc/docker//daemon.json\r\n{\r\n    \"runtimes\": {\r\n        \"nvidia\": {\r\n            \"path\": \"/usr/bin/nvidia-container-runtime\",\r\n            \"runtimeArgs\": []\r\n        }\r\n    }\r\n}\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/comments/420651248","html_url":"https://github.com/NVIDIA/k8s-device-plugin/issues/72#issuecomment-420651248","issue_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/72","id":420651248,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDY1MTI0OA==","user":{"login":"Bharathkumarraju","id":5610588,"node_id":"MDQ6VXNlcjU2MTA1ODg=","avatar_url":"https://avatars2.githubusercontent.com/u/5610588?v=4","gravatar_id":"","url":"https://api.github.com/users/Bharathkumarraju","html_url":"https://github.com/Bharathkumarraju","followers_url":"https://api.github.com/users/Bharathkumarraju/followers","following_url":"https://api.github.com/users/Bharathkumarraju/following{/other_user}","gists_url":"https://api.github.com/users/Bharathkumarraju/gists{/gist_id}","starred_url":"https://api.github.com/users/Bharathkumarraju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bharathkumarraju/subscriptions","organizations_url":"https://api.github.com/users/Bharathkumarraju/orgs","repos_url":"https://api.github.com/users/Bharathkumarraju/repos","events_url":"https://api.github.com/users/Bharathkumarraju/events{/privacy}","received_events_url":"https://api.github.com/users/Bharathkumarraju/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T13:38:23Z","updated_at":"2018-09-12T13:38:23Z","author_association":"NONE","body":"@RenaudWasTaken  Thanks for the hint able to fix it by passing kubelet args as --cluster-dns Thank you!!!"}}
{"actor":{"display_login":"Bharathkumarraju"}}