{"action":"created","issue":{"url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/167","repository_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark","labels_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/167/labels{/name}","comments_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/167/comments","events_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/167/events","html_url":"https://github.com/facebookresearch/maskrcnn-benchmark/issues/167","id":381906175,"node_id":"MDU6SXNzdWUzODE5MDYxNzU=","number":167,"title":"Docker Runtime Error: Not Compiled with GPU support","user":{"login":"archdyn","id":36210156,"node_id":"MDQ6VXNlcjM2MjEwMTU2","avatar_url":"https://avatars2.githubusercontent.com/u/36210156?v=4","gravatar_id":"","url":"https://api.github.com/users/archdyn","html_url":"https://github.com/archdyn","followers_url":"https://api.github.com/users/archdyn/followers","following_url":"https://api.github.com/users/archdyn/following{/other_user}","gists_url":"https://api.github.com/users/archdyn/gists{/gist_id}","starred_url":"https://api.github.com/users/archdyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/archdyn/subscriptions","organizations_url":"https://api.github.com/users/archdyn/orgs","repos_url":"https://api.github.com/users/archdyn/repos","events_url":"https://api.github.com/users/archdyn/events{/privacy}","received_events_url":"https://api.github.com/users/archdyn/received_events","type":"User","site_admin":false},"labels":[{"id":1103191754,"node_id":"MDU6TGFiZWwxMTAzMTkxNzU0","url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/labels/bug","name":"bug","color":"d73a4a","default":true},{"id":1104000877,"node_id":"MDU6TGFiZWwxMTA0MDAwODc3","url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/labels/contributions%20welcome","name":"contributions welcome","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-11-17T22:13:30Z","updated_at":"2018-11-19T12:50:33Z","closed_at":null,"author_association":"NONE","body":"## ❓ Questions and Help\r\nHello,\r\n\r\ni have a strange Problem with the Docker Image. When I build the Docker Image given the instructions in INSTALL.md and if I then try training on the coco2014 dataset with the command below I get RuntimeError: Not compiled with GPU support(nms at ./maskrcnn_benchmakr/csrc/nms.h:22)\r\n\r\n```\r\nnvidia-docker run --shm-size=8gb -v /home/archdyn/Datasets/coco:/maskrcnn-benchmark/datasets/coco maskrcnn-benchmark python /maskrcnn-benchmark/tools/train_net.py --config-file \"/maskrcnn-benchmark/configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS \"(480000, 640000)\" TEST.IMS_PER_BATCH 1\r\n```\r\n\r\n\r\nBut whhen I change the Dockerfile and comment the line `python setup.py build develop` before `WORKDIR /maskrcnn-benchmark` out and then execute the line `python setup.py build develop` inside my built docker container i can train without problems.\r\n\r\n\r\n\r\nMy Environment when running the Docker Container:\r\n```\r\n2018-11-17 20:03:13,889 maskrcnn_benchmark INFO: Collecting env info (might take some time)\r\n2018-11-17 20:03:15,634 maskrcnn_benchmark INFO: \r\nPyTorch version: 1.0.0.dev20181116\r\nIs debug build: No\r\nCUDA used to build PyTorch: 9.0.176\r\n\r\nOS: Ubuntu 16.04.5 LTS\r\nGCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\nCMake version: version 3.5.1\r\n\r\nPython version: 3.6\r\nIs CUDA available: Yes\r\nCUDA runtime version: 9.0.176\r\nGPU models and configuration: GPU 0: GeForce GTX 850M\r\nNvidia driver version: 410.73\r\ncuDNN version: Probably one of the following:\r\n/usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1\r\n/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a\r\n\r\nVersions of relevant libraries:\r\n[pip] Could not collect\r\n[conda] pytorch-nightly           1.0.0.dev20181116 py3.6_cuda9.0.176_cudnn7.1.2_0    pytorch\r\n        Pillow (5.3.0)\r\n```\r\n\r\nDoes somebody know why this problem happens?"},"comment":{"url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/comments/439882412","html_url":"https://github.com/facebookresearch/maskrcnn-benchmark/issues/167#issuecomment-439882412","issue_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/167","id":439882412,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTg4MjQxMg==","user":{"login":"archdyn","id":36210156,"node_id":"MDQ6VXNlcjM2MjEwMTU2","avatar_url":"https://avatars2.githubusercontent.com/u/36210156?v=4","gravatar_id":"","url":"https://api.github.com/users/archdyn","html_url":"https://github.com/archdyn","followers_url":"https://api.github.com/users/archdyn/followers","following_url":"https://api.github.com/users/archdyn/following{/other_user}","gists_url":"https://api.github.com/users/archdyn/gists{/gist_id}","starred_url":"https://api.github.com/users/archdyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/archdyn/subscriptions","organizations_url":"https://api.github.com/users/archdyn/orgs","repos_url":"https://api.github.com/users/archdyn/repos","events_url":"https://api.github.com/users/archdyn/events{/privacy}","received_events_url":"https://api.github.com/users/archdyn/received_events","type":"User","site_admin":false},"created_at":"2018-11-19T12:50:33Z","updated_at":"2018-11-19T12:50:33Z","author_association":"NONE","body":"I have a second PC at my disposal where this problem also occurs. On the second Pc I have a Nvidia V100. The Output of the second PC is below.\r\n\r\nnvcc -V:\r\n```\r\nnvcc: NVIDIA (R) Cuda compiler driver\r\nCopyright (c) 2005-2017 NVIDIA Corporation\r\nBuilt on Fri_Nov__3_21:07:56_CDT_2017\r\nCuda compilation tools, release 9.1, V9.1.85\r\n```\r\n\r\nnvidia-smi:\r\n```\r\nMon Nov 19 13:22:50 2018\r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 396.26                 Driver Version: 396.26                    |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  Tesla V100-PCIE...  Off  | 00000000:60:00.0 Off |                  Off |\r\n| N/A   32C    P0    37W / 250W |      0MiB / 16160MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n\r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n|  No running processes found                                                 |\r\n+-----------------------------------------------------------------------------+\r\n```\r\n\r\ncommand to build docker image:\r\n```\r\nnvidia-docker build -t maskrcnn-benchmark --build-arg CUDA=9.1 --build-arg CUDNN=7 docker/\r\n```\r\n\r\nPS: The order of the build command in INSTALL.md seems to be wrong. In the INSTALL.md the order is the following but this gives an error:\r\n```\r\nnvidia-docker build -t --build-arg CUDA=9.2 --build-arg CUDNN=7 maskrcnn-benchmark docker/\r\n````\r\n\r\nWhen building maskrcnn-benchmark the following Warning occurs but the build process continues and succeeds:\r\nNo CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'\r\n\r\nBut when I then train with following command I get the Runtime Error:\r\n```\r\nnvidia-docker run --shm-size=8gb -v /home/archdyn/Datasets/coco:/maskrcnn-benchmark/datasets/coco maskrcnn-benchmark python /maskrcnn-benchmark/tools/train_net.py --config-file \"/maskrcnn-benchmark/configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS \"(480000, 640000)\" TEST.IMS_PER_BATCH 1\r\n```\r\n\r\n\r\nThe only way to train and prevent the Runtime Error is to modify the Dockerfile and build it like:\r\n```\r\nARG CUDA=\"9.0\"\r\nARG CUDNN=\"7\"\r\n\r\nFROM nvidia/cuda:${CUDA}-cudnn${CUDNN}-devel-ubuntu16.04\r\n\r\nRUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections\r\n\r\n# install basics\r\nRUN apt-get update -y \\\r\n && apt-get install -y apt-utils git curl ca-certificates bzip2 cmake tree htop bmon iotop g++\r\n\r\n# Install Miniconda\r\nRUN curl -so /miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \\\r\n && chmod +x /miniconda.sh \\\r\n && /miniconda.sh -b -p /miniconda \\\r\n && rm /miniconda.sh\r\n\r\nENV PATH=/miniconda/bin:$PATH\r\n\r\n# Create a Python 3.6 environment\r\nRUN /miniconda/bin/conda install -y conda-build \\\r\n && /miniconda/bin/conda create -y --name py36 python=3.6.7 \\\r\n && /miniconda/bin/conda clean -ya\r\n\r\nENV CONDA_DEFAULT_ENV=py36\r\nENV CONDA_PREFIX=/miniconda/envs/$CONDA_DEFAULT_ENV\r\nENV PATH=$CONDA_PREFIX/bin:$PATH\r\nENV CONDA_AUTO_UPDATE_CONDA=false\r\n\r\nRUN conda install -y ipython\r\nRUN pip install ninja yacs cython matplotlib\r\n\r\n# Install PyTorch 1.0 Nightly\r\nRUN conda install -y pytorch-nightly -c pytorch && conda clean -ya\r\n\r\n# Install TorchVision master\r\nRUN git clone https://github.com/pytorch/vision.git \\\r\n && cd vision \\\r\n && python setup.py install\r\n\r\n# install pycocotools\r\nRUN git clone https://github.com/cocodataset/cocoapi.git \\\r\n && cd cocoapi/PythonAPI \\\r\n && python setup.py build_ext install\r\n\r\n# install PyTorch Detection\r\nRUN git clone https://github.com/facebookresearch/maskrcnn-benchmark.git \\\r\n\r\nWORKDIR /maskrcnn-benchmark\r\n```\r\nThen after the build I have to go inside the docker container:\r\n```\r\nnvidia-docker run --rm -it maskrcnn-benchmark bash\r\n```\r\nAnd inside the docker container I build maskrcnn-benchmark without problems:\r\n```\r\npython setup.py build develop\r\n```\r\nI then have to commit this modified docker container so that I have a Docker Image that can always be started:\r\n```\r\ndocker commit [Container ID] maskrcnn-benchmark:working\r\n```\r\nAfter all these steps I can train without problems with:\r\n```\r\nnvidia-docker run --shm-size=8gb -v /home/archdyn/Datasets/coco:/maskrcnn-benchmark/datasets/coco maskrcnn-benchmark:working python /maskrcnn-benchmark/tools/train_net.py --config-file \"/maskrcnn-benchmark/configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS \"(480000, 640000)\" TEST.IMS_PER_BATCH 1\r\n```"}}
{"actor":{"display_login":"archdyn"}}