{"action":"closed","issue":{"url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/68","repository_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin","labels_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/68/comments","events_url":"https://api.github.com/repos/NVIDIA/k8s-device-plugin/issues/68/events","html_url":"https://github.com/NVIDIA/k8s-device-plugin/issues/68","id":352648971,"node_id":"MDU6SXNzdWUzNTI2NDg5NzE=","number":68,"title":"Setting nvidia.com/gpu:NoSchedule taint causes GPU nodes to repel k8s-device-plugin, making GPU nodes unschedulable for GPU jobs","user":{"login":"notnami","id":1631585,"node_id":"MDQ6VXNlcjE2MzE1ODU=","avatar_url":"https://avatars1.githubusercontent.com/u/1631585?v=4","gravatar_id":"","url":"https://api.github.com/users/notnami","html_url":"https://github.com/notnami","followers_url":"https://api.github.com/users/notnami/followers","following_url":"https://api.github.com/users/notnami/following{/other_user}","gists_url":"https://api.github.com/users/notnami/gists{/gist_id}","starred_url":"https://api.github.com/users/notnami/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/notnami/subscriptions","organizations_url":"https://api.github.com/users/notnami/orgs","repos_url":"https://api.github.com/users/notnami/repos","events_url":"https://api.github.com/users/notnami/events{/privacy}","received_events_url":"https://api.github.com/users/notnami/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-08-21T18:01:31Z","updated_at":"2018-08-22T22:49:50Z","closed_at":"2018-08-22T22:49:50Z","author_association":"NONE","body":"**The original issue described below shows a symptom, but not the cause of the problem. For the underlying cause, please see further comments.**\r\n\r\nHi everyone,\r\n\r\nI'm sure this will be something obvious in retrospect, but I've been stuck with this for a while.\r\n\r\nAs the title says, my GPU node is reporting itself as having `alpha.kubernetes.io/nvidia-gpu` instead of `nvidia.com/gpu`, where previously it had reported `nvidia.com/gpu`. As a result, my pods, which request `nvidia.com/gpu` don't get scheduled.\r\n\r\nHere is the relevant section of `kubectl describe node`:\r\n\r\n```\r\nCapacity:\r\n alpha.kubernetes.io/nvidia-gpu:  1\r\n cpu:                             4\r\n ephemeral-storage:               130046416Ki\r\n hugepages-1Gi:                   0\r\n hugepages-2Mi:                   0\r\n memory:                          62879988Ki\r\n pods:                            110\r\nAllocatable:\r\n alpha.kubernetes.io/nvidia-gpu:  1\r\n cpu:                             4\r\n ephemeral-storage:               119850776788\r\n hugepages-1Gi:                   0\r\n hugepages-2Mi:                   0\r\n memory:                          62777588Ki\r\n pods:                            110\r\nSystem Info:\r\n Machine ID:                 6b08f5e8c7ae4843a46ae7f889a4078a\r\n System UUID:                EC292D81-8715-3D89-A766-3E173EAC5036\r\n Boot ID:                    3ed3f5ec-1ddf-43b4-9235-bf8fe261a9d3\r\n Kernel Version:             4.4.0-1065-aws\r\n OS Image:                   Ubuntu 16.04.5 LTS\r\n Operating System:           linux\r\n Architecture:               amd64\r\n Container Runtime Version:  docker://17.3.2\r\n Kubelet Version:            v1.10.7\r\n Kube-Proxy Version:         v1.10.7\r\n```\r\n\r\nI had the GPU nodes working earlier. I am lost as to what exactly changed, but before this stopped working, I added a Taint so that pods that don't need a GPU would not get scheduled on this node.\r\n\r\nFor context, I am running the cluster with `kops` on AWS. I am using k8s-device-plugin v1.10. I know the recommended version of k8s and the plug-in is 1.11, but I know it has worked before with earlier versions, so I expect it should work with 1.10\r\n\r\nI have tried re-setting the node afresh, but to no avail. I have also upgraded my cluster's k8s version to 1.10 from 1.9, but also to no result. Any help would be appreciated!"}}
{"actor":{"display_login":"flx42"}}