title: How to solve the famous `unhandled cuda error, NCCL version 2.7.8` error?
body: <p>I've seen multiple issue about the:</p>
  <pre><code>RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378083779/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
  ncclUnhandledCudaError: Call to CUDA function failed.
  </code></pre>
  <p>but none seem to fix it for me:</p>
  <ul>
  <li><a href="https://github.com/pytorch/pytorch/issues/54550" rel="noreferrer">https://github.com/pytorch/pytorch/issues/54550</a></li>
  <li><a href="https://github.com/pytorch/pytorch/issues/47885" rel="noreferrer">https://github.com/pytorch/pytorch/issues/47885</a></li>
  <li><a href="https://github.com/pytorch/pytorch/issues/50921" rel="noreferrer">https://github.com/pytorch/pytorch/issues/50921</a></li>
  <li><a href="https://github.com/pytorch/pytorch/issues/54823" rel="noreferrer">https://github.com/pytorch/pytorch/issues/54823</a></li>
  </ul>
  <p>I've tried to do <code>torch.cuda.set_device(device)</code> manually at the beginning of every script. That didn't seem to work for me. I've tried different GPUS. I've tried downgrading pytorch version and cuda version. Different combinations of 1.6.0, 1.7.1, 1.8.0 and cuda 10.2, 11.0, 11.1. I am unsure what else to do. What did people do to solve this issue?</p>
  <hr />
  <p>very related perhaps?</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/61075390/pytorch-nccl-error-unhandled-system-error-nccl-version-2-4-8">Pytorch &quot;NCCL error&quot;: unhandled system error, NCCL version 2.4.8&quot;</a></li>
  </ul>
  <hr />
  <p>More complete error message:</p>
  <pre><code>('jobid', 4852)
  ('slurm_jobid', -1)
  ('slurm_array_task_id', -1)
  ('condor_jobid', 4852)
  ('current_time', 'Mar25_16-27-35')
  ('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar25_16-27-35_jobid_4852/tb'))
  ('gpu_name', 'GeForce GTX TITAN X')
  ('PID', '30688')
  torch.cuda.device_count()=2
  
  opts.world_size=2
  
  ABOUT TO SPAWN WORKERS
  done setting sharing strategy...next mp.spawn
  INFO:root:Added key: store_based_barrier_key:1 to store for rank: 1
  INFO:root:Added key: store_based_barrier_key:1 to store for rank: 0
  rank=0
  mp.current_process()=&lt;SpawnProcess name='SpawnProcess-1' parent=30688 started&gt;
  os.getpid()=30704
  setting up rank=0 (with world_size=2)
  MASTER_ADDR='127.0.0.1'
  59264
  backend='nccl'
  --&gt; done setting up rank=0
  setup process done for rank=0
  Traceback (most recent call last):
    File &quot;/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py&quot;, line 279, in &lt;module&gt;
      main_distributed()
    File &quot;/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py&quot;, line 188, in main_distributed
      spawn_return = mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 230, in spawn
      return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 188, in start_processes
      while not context.join():
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 150, in join
      raise ProcessRaisedException(msg, error_index, failed_process.pid)
  torch.multiprocessing.spawn.ProcessRaisedException: 
  
  -- Process 0 terminated with the following error:
  Traceback (most recent call last):
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 59, in _wrap
      fn(i, *args)
    File &quot;/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py&quot;, line 212, in train
      tactic_predictor = move_to_ddp(rank, opts, tactic_predictor)
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-project/uutils/torch/distributed.py&quot;, line 162, in move_to_ddp
      model = DistributedDataParallel(model, find_unused_parameters=True, device_ids=[opts.gpu])
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py&quot;, line 446, in __init__
      self._sync_params_and_buffers(authoritative_rank=0)
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py&quot;, line 457, in _sync_params_and_buffers
      self._distributed_broadcast_coalesced(
    File &quot;/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py&quot;, line 1155, in _distributed_broadcast_coalesced
      dist._broadcast_coalesced(
  RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1616554793803/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
  ncclUnhandledCudaError: Call to CUDA function failed.
  </code></pre>
  <hr />
  <h1>Bonus 1:</h1>
  <p>I still have errors:</p>
  <pre><code>ncclSystemError: System call (socket, malloc, munmap, etc) failed.
  Traceback (most recent call last):
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1423, in &lt;module&gt;
      main()
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1365, in main
      train(args=args)
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1385, in train
      args.opt = move_opt_to_cherry_opt_and_sync_params(args) if is_running_parallel(args.rank) else args.opt
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/distributed.py&quot;, line 456, in move_opt_to_cherry_opt_and_sync_params
      args.opt = cherry.optim.Distributed(args.model.parameters(), opt=args.opt, sync=syn)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py&quot;, line 62, in __init__
      self.sync_parameters()
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py&quot;, line 78, in sync_parameters
      dist.broadcast(p.data, src=root)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1090, in broadcast
      work = default_pg.broadcast([tensor], opts)
  RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8
  </code></pre>
  <p>one of the answers suggested to have nvcca &amp; pytorch.version.cuda to match but they do not:</p>
  <pre><code>(meta_learning_a100) [miranda9@hal-dgx ~]$ python -c &quot;import torch;print(torch.version.cuda)&quot;
  
  11.1
  (meta_learning_a100) [miranda9@hal-dgx ~]$ nvcc -V
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2020 NVIDIA Corporation
  Built on Wed_Jul_22_19:09:09_PDT_2020
  Cuda compilation tools, release 11.0, V11.0.221
  Build cuda_11.0_bu.TC445_37.28845127_0
  </code></pre>
  <p>How do I match them?</p>