{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/831","repository_url":"https://api.github.com/repos/NVIDIA/nvidia-docker","labels_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/831/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/831/comments","events_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/831/events","html_url":"https://github.com/NVIDIA/nvidia-docker/issues/831","id":364661775,"node_id":"MDU6SXNzdWUzNjQ2NjE3NzU=","number":831,"title":"Nvidia-docker containers fail to initialize with a CUDA error: out of memory","user":{"login":"dtrash1","id":5915141,"node_id":"MDQ6VXNlcjU5MTUxNDE=","avatar_url":"https://avatars2.githubusercontent.com/u/5915141?v=4","gravatar_id":"","url":"https://api.github.com/users/dtrash1","html_url":"https://github.com/dtrash1","followers_url":"https://api.github.com/users/dtrash1/followers","following_url":"https://api.github.com/users/dtrash1/following{/other_user}","gists_url":"https://api.github.com/users/dtrash1/gists{/gist_id}","starred_url":"https://api.github.com/users/dtrash1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtrash1/subscriptions","organizations_url":"https://api.github.com/users/dtrash1/orgs","repos_url":"https://api.github.com/users/dtrash1/repos","events_url":"https://api.github.com/users/dtrash1/events{/privacy}","received_events_url":"https://api.github.com/users/dtrash1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-27T21:33:08Z","updated_at":"2018-09-28T18:40:55Z","closed_at":null,"author_association":"NONE","body":"### 1. Issue or feature description\r\nNvidia-docker containers always fail to initialize with a CUDA error: out of memory.  The docker install seems to be OK in that it can run non-nvidia images successfully (i.e. the hello-world image works).\r\n### 2. Steps to reproduce the issue\r\n1. Install using the Quickstart section of README.md\r\n2. Upon running the final (test) step:\r\n```\r\n$ docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi\r\n```\r\n Receive the error:\r\n```\r\ndocker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 1 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.0 --pid=6610 /var/lib/docker/overlay2/e13075e398b2d91aee86c6a26fab25cb44c09987b6b099f09e71f16746a6b53d/merged]\\\\\\\\nnvidia-container-cli: initialization error: cuda error: out of memory\\\\\\\\n\\\\\\\"\\\"\": unknown.\r\n```\r\nI have 3 1080Ti cards and one NVS310 that drives the display.  setting `NVIDIA_VISIBLE_DEVICES=2` (for example) does change the `--device=` argument for `nvidia-container-cli` but the returned error is essentially identical.\r\n\r\nI'm not really sure how to proceed with troubleshooting, and haven't found anyone else with a similar problem.\r\n\r\n### 3. Information to [attach](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/) (optional if deemed irrelevant)\r\n\r\n - [x] Kernel version from `uname -a`\r\n```\r\nLinux inslindl2 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n - [ ] Any relevant kernel output lines from `dmesg`\r\n - [x] Driver information from `nvidia-smi -a`\r\nThe basic info:\r\n```\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  NVS 310             Off  | 00000000:05:00.0 N/A |                  N/A |\r\n| 30%   56C    P0    N/A /  N/A |    244MiB /   451MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |\r\n| 28%   42C    P8    10W / 250W |      2MiB / 11178MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   2  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |\r\n| 28%   39C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   3  GeForce GTX 108...  Off  | 00000000:0A:00.0 Off |                  N/A |\r\n|  0%   41C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n|    0                    Not Supported                                       |\r\n+-----------------------------------------------------------------------------+\r\n\r\n```\r\nAnd the whole output:\r\n[nvidia-smi.out.txt](https://github.com/NVIDIA/nvidia-docker/files/2425943/nvidia-smi.out.txt)\r\n - [x] Docker version from `docker version`\r\n```\r\nClient:\r\n Version:           18.06.1-ce\r\n API version:       1.38\r\n Go version:        go1.10.3\r\n Git commit:        e68fc7a\r\n Built:             Tue Aug 21 17:24:51 2018\r\n OS/Arch:           linux/amd64\r\n Experimental:      false\r\n\r\nServer:\r\n Engine:\r\n  Version:          18.06.1-ce\r\n  API version:      1.38 (minimum version 1.12)\r\n  Go version:       go1.10.3\r\n  Git commit:       e68fc7a\r\n  Built:            Tue Aug 21 17:23:15 2018\r\n  OS/Arch:          linux/amd64\r\n  Experimental:     false\r\n```\r\n - [x] NVIDIA packages version from `dpkg -l '*nvidia*'` _or_ `rpm -qa '*nvidia*'`\r\n```\r\nDesired=Unknown/Install/Remove/Purge/Hold\r\n| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend\r\n|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)\r\n||/ Name           Version      Architecture Description\r\n+++-==============-============-============-=================================\r\nun  libgldispatch0 <none>       <none>       (no description available)\r\nii  libnvidia-cfg1 390.48-0ubun amd64        NVIDIA binary OpenGL/GLX configur\r\nun  libnvidia-cfg1 <none>       <none>       (no description available)\r\nun  libnvidia-comm <none>       <none>       (no description available)\r\nii  libnvidia-comm 390.48-0ubun all          Shared files used by the NVIDIA l\r\nii  libnvidia-comp 390.48-0ubun amd64        NVIDIA libcompute package\r\nii  libnvidia-comp 390.48-0ubun i386         NVIDIA libcompute package\r\nii  libnvidia-cont 1.0.0-1      amd64        NVIDIA container runtime library \r\nii  libnvidia-cont 1.0.0-1      amd64        NVIDIA container runtime library\r\nun  libnvidia-deco <none>       <none>       (no description available)\r\nii  libnvidia-deco 390.48-0ubun amd64        NVIDIA Video Decoding runtime lib\r\nii  libnvidia-deco 390.48-0ubun i386         NVIDIA Video Decoding runtime lib\r\nun  libnvidia-enco <none>       <none>       (no description available)\r\nii  libnvidia-enco 390.48-0ubun amd64        NVENC Video Encoding runtime libr\r\nii  libnvidia-enco 390.48-0ubun i386         NVENC Video Encoding runtime libr\r\nun  libnvidia-fbc1 <none>       <none>       (no description available)\r\nii  libnvidia-fbc1 390.48-0ubun amd64        NVIDIA OpenGL-based Framebuffer C\r\nii  libnvidia-fbc1 390.48-0ubun i386         NVIDIA OpenGL-based Framebuffer C\r\nun  libnvidia-gl   <none>       <none>       (no description available)\r\nii  libnvidia-gl-3 390.48-0ubun amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND \r\nii  libnvidia-gl-3 390.48-0ubun i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND \r\nun  libnvidia-ifr1 <none>       <none>       (no description available)\r\nii  libnvidia-ifr1 390.48-0ubun amd64        NVIDIA OpenGL-based Inband Frame \r\nii  libnvidia-ifr1 390.48-0ubun i386         NVIDIA OpenGL-based Inband Frame \r\nun  nvidia-304     <none>       <none>       (no description available)\r\nun  nvidia-340     <none>       <none>       (no description available)\r\nun  nvidia-384     <none>       <none>       (no description available)\r\nun  nvidia-390     <none>       <none>       (no description available)\r\nun  nvidia-common  <none>       <none>       (no description available)\r\nii  nvidia-compute 390.48-0ubun amd64        NVIDIA compute utilities\r\nii  nvidia-contain 2.0.0+docker amd64        NVIDIA container runtime\r\nii  nvidia-contain 1.4.0-1      amd64        NVIDIA container runtime hook\r\nii  nvidia-dkms-39 390.48-0ubun amd64        NVIDIA DKMS package\r\nun  nvidia-dkms-ke <none>       <none>       (no description available)\r\nun  nvidia-docker  <none>       <none>       (no description available)\r\nii  nvidia-docker2 2.0.3+docker all          nvidia-docker CLI wrapper\r\nii  nvidia-driver- 390.48-0ubun amd64        NVIDIA driver metapackage\r\nun  nvidia-driver- <none>       <none>       (no description available)\r\nun  nvidia-kernel- <none>       <none>       (no description available)\r\nii  nvidia-kernel- 390.48-0ubun amd64        Shared files used with the kernel\r\nun  nvidia-kernel- <none>       <none>       (no description available)\r\nii  nvidia-kernel- 390.48-0ubun amd64        NVIDIA kernel source package\r\nun  nvidia-legacy- <none>       <none>       (no description available)\r\nun  nvidia-libopen <none>       <none>       (no description available)\r\nun  nvidia-opencl- <none>       <none>       (no description available)\r\nun  nvidia-persist <none>       <none>       (no description available)\r\nii  nvidia-prime   0.8.8        all          Tools to enable NVIDIA's Prime\r\nii  nvidia-setting 390.42-0ubun amd64        Tool for configuring the NVIDIA g\r\nun  nvidia-setting <none>       <none>       (no description available)\r\nun  nvidia-smi     <none>       <none>       (no description available)\r\nun  nvidia-utils   <none>       <none>       (no description available)\r\nii  nvidia-utils-3 390.48-0ubun amd64        NVIDIA driver support binaries\r\nun  nvidia-vdpau-d <none>       <none>       (no description available)\r\nii  xserver-xorg-v 390.48-0ubun amd64        NVIDIA binary Xorg driver\r\n\r\n```\r\n - [x] NVIDIA container library version from `nvidia-container-cli -V`\r\n```\r\nversion: 1.0.0\r\nbuild date: 2018-09-20T20:19+00:00\r\nbuild revision: 881c88e2e5bb682c9bb14e68bd165cfb64563bb1\r\nbuild compiler: x86_64-linux-gnu-gcc-7 7.3.0\r\nbuild platform: x86_64\r\nbuild flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections\r\n```\r\n - [x] NVIDIA container library logs (see [troubleshooting](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting))\r\nFrom `/var/log/nvidia-container-runtime-hook.log`\r\n```\r\nI0927 21:25:32.311741 7298 nvc.c:281] initializing library context (version=1.0.0, build=881c88e2e5bb682c9bb14e68bd165cfb64563bb1)\r\nI0927 21:25:32.311821 7298 nvc.c:255] using root /\r\nI0927 21:25:32.311833 7298 nvc.c:256] using ldcache /etc/ld.so.cache\r\nI0927 21:25:32.311843 7298 nvc.c:257] using unprivileged user 65534:65534\r\nI0927 21:25:32.314196 7304 nvc.c:191] loading kernel module nvidia\r\nI0927 21:25:32.314496 7304 nvc.c:203] loading kernel module nvidia_uvm\r\nI0927 21:25:32.314652 7304 nvc.c:211] loading kernel module nvidia_modeset\r\nI0927 21:25:32.314998 7305 driver.c:133] starting driver service\r\nI0927 21:25:32.403102 7298 driver.c:233] driver service terminated with signal 15\r\n```\r\n - [x] Docker command, image and tag used\r\n```\r\n$ docker run --runtime=nvidia --env NVIDIA_VISIBLE_DEVICES=2 --rm nvidia/cuda:9.0-base bash\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/comments/425529231","html_url":"https://github.com/NVIDIA/nvidia-docker/issues/831#issuecomment-425529231","issue_url":"https://api.github.com/repos/NVIDIA/nvidia-docker/issues/831","id":425529231,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTUyOTIzMQ==","user":{"login":"3XX0","id":3028125,"node_id":"MDQ6VXNlcjMwMjgxMjU=","avatar_url":"https://avatars2.githubusercontent.com/u/3028125?v=4","gravatar_id":"","url":"https://api.github.com/users/3XX0","html_url":"https://github.com/3XX0","followers_url":"https://api.github.com/users/3XX0/followers","following_url":"https://api.github.com/users/3XX0/following{/other_user}","gists_url":"https://api.github.com/users/3XX0/gists{/gist_id}","starred_url":"https://api.github.com/users/3XX0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3XX0/subscriptions","organizations_url":"https://api.github.com/users/3XX0/orgs","repos_url":"https://api.github.com/users/3XX0/repos","events_url":"https://api.github.com/users/3XX0/events{/privacy}","received_events_url":"https://api.github.com/users/3XX0/received_events","type":"User","site_admin":false},"created_at":"2018-09-28T18:40:55Z","updated_at":"2018-09-28T18:40:55Z","author_association":"MEMBER","body":"@dtrash1 you can exclude your NVS card see if it works:\r\n\r\n```\r\nsudo tee /etc/systemd/system/docker.service.d/override.conf <<EOF\r\n[Service]\r\nExecStart=\r\nExecStart=/usr/bin/dockerd --host=fd://\r\nEOF\r\n\r\nnvidia-container-cli list -d 1,2,3 | sed 's/.*/DeviceAllow=& rwm/' | sudo tee -a /etc/systemd/system/docker.service.d/override.conf\r\n\r\nsudo systemctl restart docker\r\n```"}}
{"actor":{"display_login":"3XX0"}}