{"action":"created","issue":{"url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/259","repository_url":"https://api.github.com/repos/denizyuret/Knet.jl","labels_url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/259/labels{/name}","comments_url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/259/comments","events_url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/259/events","html_url":"https://github.com/denizyuret/Knet.jl/issues/259","id":288951491,"number":259,"title":"Knet not respecting CUDA_VISIBLE_DEVICES?","user":{"login":"davidavdav","id":5497303,"avatar_url":"https://avatars0.githubusercontent.com/u/5497303?v=4","gravatar_id":"","url":"https://api.github.com/users/davidavdav","html_url":"https://github.com/davidavdav","followers_url":"https://api.github.com/users/davidavdav/followers","following_url":"https://api.github.com/users/davidavdav/following{/other_user}","gists_url":"https://api.github.com/users/davidavdav/gists{/gist_id}","starred_url":"https://api.github.com/users/davidavdav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidavdav/subscriptions","organizations_url":"https://api.github.com/users/davidavdav/orgs","repos_url":"https://api.github.com/users/davidavdav/repos","events_url":"https://api.github.com/users/davidavdav/events{/privacy}","received_events_url":"https://api.github.com/users/davidavdav/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-01-16T15:38:16Z","updated_at":"2018-02-14T07:42:35Z","closed_at":null,"author_association":"NONE","body":"Hello, \r\n\r\nI have trouble running Knet under a Slurm resource manager.  I understand that Slurm sets the environment variable `CUDA_VISIBLE_DEVICES` to a list of devices available to the process that is spawned by Slurm.   In a configuration with 2 GPUs per node, I \r\n - can run a Knet job (the char-rnn example), when I request Slurm `--gres gpu:2`, resulting in `ENV{\"CUDA_VISIBLE_DEVICES\"}==\"0,1\"`, and Knet uses only 1 of the allotted 2 GPUs\r\n - can *not* run a Knet job, when I request Slurm `--gres gpu:1`, resulting in `ENV{\"CUDA_VISIBLE_DEVICES\"}==\"0\"`, with error message\r\n```julia\r\nERROR: LoadError: KnetPtr: bad device id -1.\r\nStacktrace:\r\n [1] Knet.KnetPtr(::Int64) at /home/david/.julia/v0.6/Knet/src/kptr.jl:73\r\n```\r\nIn the second case, `Knet.gpuCount() == 2` (which should be 1, because only one GPU is assigned to the Julia process).  "},"comment":{"url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/comments/365521464","html_url":"https://github.com/denizyuret/Knet.jl/issues/259#issuecomment-365521464","issue_url":"https://api.github.com/repos/denizyuret/Knet.jl/issues/259","id":365521464,"user":{"login":"cangumeli","id":9155560,"avatar_url":"https://avatars1.githubusercontent.com/u/9155560?v=4","gravatar_id":"","url":"https://api.github.com/users/cangumeli","html_url":"https://github.com/cangumeli","followers_url":"https://api.github.com/users/cangumeli/followers","following_url":"https://api.github.com/users/cangumeli/following{/other_user}","gists_url":"https://api.github.com/users/cangumeli/gists{/gist_id}","starred_url":"https://api.github.com/users/cangumeli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cangumeli/subscriptions","organizations_url":"https://api.github.com/users/cangumeli/orgs","repos_url":"https://api.github.com/users/cangumeli/repos","events_url":"https://api.github.com/users/cangumeli/events{/privacy}","received_events_url":"https://api.github.com/users/cangumeli/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T07:42:35Z","updated_at":"2018-02-14T07:42:35Z","author_association":"COLLABORATOR","body":"Do you have nvml working properly?  You can check this simply by evaluating `Knet.nvmlfound`. If nvml doesn't exist, regular cuda API is used. This may cause the problem."}}
{"actor":{"display_login":"cangumeli"}}