title: CUDNN_STATUS_MAPPING_ERROR when training with pose2body
body: <p>I'm trying to train <a href="https://github.com/NVIDIA/vid2vid" rel="nofollow noreferrer">https://github.com/NVIDIA/vid2vid</a>. I'm...</p>
  
  <ul>
  <li><p>...executing with pretty much the vanilla parametrization shown in the readme, I had to change the number of GPUs though and increased the number of threads for reading the dataset. Command:</p>
  
  <p><code>python train.py \
  --name pose2body_256p  \
  --dataroot datasets/pose  \
  --dataset_mode pose  \
  --input_nc 6  \
  --num_D 2  \ 
  --resize_or_crop ScaleHeight_and_scaledCrop  \
  --loadSize 384  \
  --fineSize 256  \
  --gpu_ids 0,1  \
  --batchSize 1  \
  --max_frames_per_gpu 3  \
  --no_first_img  \
  --n_frames_total 12  \
  --max_t_step 4  \
  --nThreads 6</code> </p></li>
  <li><p>...training on the supplied example datasets.</p></li>
  <li>...running a docker container built with the scripts in <code>vid2vid/docker</code>, e. g. with CUDA 9.0 and CUDNN 7.</li>
  <li>...using two NVIDIA V100 GPUs.</li>
  </ul>
  
  <p>Whenever I start training the script crashes after a couple of minutes with the message <strong><code>RuntimeError: CUDNN_STATUS_MAPPING_ERROR</code></strong>. Full error message:</p>
  
  <pre><code>Traceback (most recent call last):
    File "train.py", line 329, in &lt;module&gt;
      train()
    File "train.py", line 104, in train
      fake_B, fake_B_raw, flow, weight, real_A, real_Bp, fake_B_last = modelG(input_A, input_B, inst_A, fake_B_last)            
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
      outputs = self.parallel_apply(replicas, inputs, kwargs)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
      return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
      raise output
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
      output = module(*input, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
      result = self.forward(*input, **kwargs)
    File "/vid2vid/models/vid2vid_model_G.py", line 130, in forward
      fake_B, fake_B_raw, flow, weight = self.generate_frame_train(netG, real_A_all, fake_B_prev, start_gpu, is_first_frame)        
    File "/vid2vid/models/vid2vid_model_G.py", line 175, in generate_frame_train
      fake_B_feat, flow_feat, fake_B_fg_feat, use_raw_only)
    File "/vid2vid/models/networks.py", line 171, in forward
      downsample = self.model_down_seg(input) + self.model_down_img(img_prev)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 91, in forward
      input = module(input)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
      result = self.forward(*input, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
      self.padding, self.dilation, self.groups)
  RuntimeError: CUDNN_STATUS_MAPPING_ERROR
  </code></pre>
  
  <p>From reading the issues in the vid2vid using two V100 should work with this setup. The error also occurs if CUDA 8/CUDNN 6 are used. I checked the flags but haven't found any indication of further necessary changes to the arguments supplied to <code>train.py</code>.</p>
  
  <p>Any ideas on how to solve (or work around) this?</p>