{"action":"opened","issue":{"url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/216","repository_url":"https://api.github.com/repos/NVIDIA/nvidia-docker","labels_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/216/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/216/comments","events_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/216/events","html_url":"https://github.com/NVIDIA/nvidia-docker/issues/216","id":182162613,"number":216,"title":"docker: Error response from daemon: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details.","user":{"login":"jjgraham","id":11588369,"avatar_url":"https://avatars.githubusercontent.com/u/11588369?v=3","gravatar_id":"","url":"https://api.github.com/users/jjgraham","html_url":"https://github.com/jjgraham","followers_url":"https://api.github.com/users/jjgraham/followers","following_url":"https://api.github.com/users/jjgraham/following{/other_user}","gists_url":"https://api.github.com/users/jjgraham/gists{/gist_id}","starred_url":"https://api.github.com/users/jjgraham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jjgraham/subscriptions","organizations_url":"https://api.github.com/users/jjgraham/orgs","repos_url":"https://api.github.com/users/jjgraham/repos","events_url":"https://api.github.com/users/jjgraham/events{/privacy}","received_events_url":"https://api.github.com/users/jjgraham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-11T03:04:23Z","updated_at":"2016-10-11T03:04:23Z","closed_at":null,"body":"I am trying to install nvidia-docker on a machine with 2x M40 GPUs running Centos 7.2 with a fresh yum update. I completely removed all hints of docker anything on the server before starting.\r\n\r\nI have 2x 1TB NVMe devices formatted btrfs and mounted on /var/lib/docker and /var/lib/nvidia-docker. The install of docker and nvidia docker looks fine but when i try to run nvidia-smi in a container it barfs.\r\n\r\nI can talk to the container via docker \r\n[root@patternlab ~]# docker run --rm nvidia/cuda cat  /etc/debian_version\r\njessie/sid\r\n\r\nbut can't using nvidia-docker\r\n[root@patternlab ~]# nvidia-docker run --rm nvidia/cuda cat  /etc/debian_version\r\ndocker: Error response from daemon: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details.\r\nSee 'docker run --help'.\r\n\r\n/var/log/messages shows:\r\nOct 10 18:46:05 patternlab dockerd: time=\"2016-10-10T18:46:05.194777571-07:00\" level=error msg=\"Handler for GET /v1.24/volumes/nvidia_driver_361.93.02 returned error: get nvidia_driver_361.93.02: no such volume\"\r\n\r\nThe only thing i see strange is /var/lib/nvidia-docker/volumes/nvidia_driver is empty\r\n\r\n[root@patternlab ~]# ls -al /usr/bin/nvidia-cuda-mps-control /var/lib/nvidia-docker/volumes/nvidia_driver\r\n-rwxr-xr-x. 1 root          root          59000 Oct  6 18:35 /usr/bin/nvidia-cuda-mps-control\r\n\r\n/var/lib/nvidia-docker/volumes/nvidia_driver:\r\ntotal 0\r\ndrwxr-xr-x. 1 nvidia-docker nvidia-docker  0 Oct 10 18:50 .\r\ndrwxr-xr-x. 1 nvidia-docker nvidia-docker 26 Oct  9 11:46 ..\r\n\r\n[root@patternlab ~]# nvidia-docker run --rm nvidia/cuda nvidia-smi\r\ndocker: Error response from daemon: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details.\r\nSee 'docker run --help'.\r\n\r\n[root@patternlab ~]# rpm -qa | grep docker\r\ndocker-engine-1.12.1-1.el7.centos.x86_64\r\ndocker-engine-selinux-1.12.1-1.el7.centos.noarch\r\nnvidia-docker-1.0.0~rc.3-1.x86_64\r\n\r\n[root@patternlab ~]# docker version\r\nClient:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n\r\n[root@patternlab ~]# docker info\r\nContainers: 39\r\n Running: 0\r\n Paused: 0\r\n Stopped: 39\r\nImages: 129\r\nServer Version: 1.12.1\r\nStorage Driver: btrfs\r\n Build Version: Btrfs v3.19.1\r\n Library Version: 101\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null bridge host overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 3.10.0-327.36.1.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 40\r\nTotal Memory: 125.8 GiB\r\nName: patternlab...\r\nID: BRCR:FEZ7:E5MS:AQRE:B7MR:HUV7:KPTC:MH4Q:2KP2:AHP4:ZWLO:EOMO\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-iptables is disabled\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n\r\nThis is /var/log/messages showing startup and failure when a container is run\r\n\r\nOct 10 18:45:09 patternlab systemd: Starting NVIDIA Docker plugin...\r\nOct 10 18:45:09 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:09 Loading NVIDIA unified memory\r\nOct 10 18:45:09 patternlab kernel: nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 243\r\nOct 10 18:45:09 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:09 Loading NVIDIA management library\r\nOct 10 18:45:09 patternlab systemd: Started NVIDIA Docker plugin.\r\nOct 10 18:45:17 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:17 Discovering GPU devices\r\nOct 10 18:45:18 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:18 Provisioning volumes at /var/lib/nvidia-docker/volumes\r\nOct 10 18:45:18 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:18 Serving plugin API at /var/lib/nvidia-docker\r\nOct 10 18:45:18 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:45:18 Serving remote API at localhost:3476\r\nOct 10 18:46:05 patternlab dockerd: time=\"2016-10-10T18:46:05.121675274-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:46:05 patternlab dockerd: time=\"2016-10-10T18:46:05.170028283-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:46:05 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:46:05 Received activate request\r\nOct 10 18:46:05 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:46:05 Plugins activated [VolumeDriver]\r\nOct 10 18:46:05 patternlab dockerd: time=\"2016-10-10T18:46:05.194777571-07:00\" level=error msg=\"Handler for GET /v1.24/volumes/nvidia_driver_361.93.02 returned error: get nvidia_driver_361.93.02: no such volume\"\r\nOct 10 18:46:05 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:46:05 Received create request for volume 'nvidia_driver_361.93.02'\r\nOct 10 18:46:05 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:46:05 Error: link /usr/bin/nvidia-cuda-mps-control /var/lib/nvidia-docker/volumes/nvidia_driver/361.93.02/bin/nvidia-cuda-mps-control: invalid cross-device link\r\nOct 10 18:46:06 patternlab dockerd: time=\"2016-10-10T18:46:06.103073247-07:00\" level=error msg=\"Handler for POST /v1.24/containers/create returned error: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details\\n\"\r\nOct 10 18:50:07 patternlab dockerd: time=\"2016-10-10T18:50:07.500706506-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:50:07 patternlab dockerd: time=\"2016-10-10T18:50:07.546354707-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:50:07 patternlab dockerd: time=\"2016-10-10T18:50:07.565630827-07:00\" level=error msg=\"Handler for GET /v1.24/volumes/nvidia_driver_361.93.02 returned error: get nvidia_driver_361.93.02: no such volume\"\r\nOct 10 18:50:07 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:50:07 Received create request for volume 'nvidia_driver_361.93.02'\r\nOct 10 18:50:07 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:50:07 Error: link /usr/bin/nvidia-cuda-mps-control /var/lib/nvidia-docker/volumes/nvidia_driver/361.93.02/bin/nvidia-cuda-mps-control: invalid cross-device link\r\nOct 10 18:50:08 patternlab dockerd: time=\"2016-10-10T18:50:08.248957378-07:00\" level=error msg=\"Handler for POST /v1.24/containers/create returned error: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details\\n\"\r\nOct 10 18:50:19 patternlab dockerd: time=\"2016-10-10T18:50:19.124472272-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:50:19 patternlab dockerd: time=\"2016-10-10T18:50:19.171098610-07:00\" level=error msg=\"Handler for GET /v1.24/containers/nvidia/cuda/json returned error: No such container: nvidia/cuda\"\r\nOct 10 18:50:19 patternlab dockerd: time=\"2016-10-10T18:50:19.196644375-07:00\" level=error msg=\"Handler for GET /v1.24/volumes/nvidia_driver_361.93.02 returned error: get nvidia_driver_361.93.02: no such volume\"\r\nOct 10 18:50:19 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:50:19 Received create request for volume 'nvidia_driver_361.93.02'\r\nOct 10 18:50:19 patternlab nvidia-docker-plugin: /usr/bin/nvidia-docker-plugin | 2016/10/10 18:50:19 Error: link /usr/bin/nvidia-cuda-mps-control /var/lib/nvidia-docker/volumes/nvidia_driver/361.93.02/bin/nvidia-cuda-mps-control: invalid cross-device link\r\nOct 10 18:50:19 patternlab dockerd: time=\"2016-10-10T18:50:19.875520810-07:00\" level=error msg=\"Handler for POST /v1.24/containers/create returned error: create nvidia_driver_361.93.02: VolumeDriver.Create: internal error, check logs for details\\n\""}}
{"actor":{"display_login":"jjgraham"}}