title: How to run torch.distributed.run such that one can run two instances of the script without errors or deadlocks?
body: <p>when I run</p>
  <pre><code>python -m torch.distributed.run --nproc_per_node=8 --master_addr=&quot;127.0.0.1&quot; --master_port=$RANDOM ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py --manual_loads_name l2l_resnet12rfs_cifarfs_adam_cl_80k
  </code></pre>
  <p>I get the error:</p>
  <pre><code>====&gt; about to start train loop
  Starting training!
  Traceback (most recent call last):
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/algorithms/maml.py&quot;, line 159, in adapt
      gradients = grad(loss,
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/autograd/__init__.py&quot;, line 226, in grad
      return Variable._execution_engine.run_backward(
  RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 6; 39.59 GiB total capacity; 265.23 MiB already allocated; 10.19 MiB free; 282.00 MiB reserved in total by PyTorch)
  learn2learn: Maybe try with allow_nograd=True and/or allow_unused=True ?
  Traceback (most recent call last):
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 216, in &lt;module&gt;
      main()
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 161, in main
      train(args=args)
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 196, in train
      meta_train_iterations_ala_l2l(args, args.agent, args.opt, args.scheduler)
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/training/meta_training.py&quot;, line 149, in meta_train_iterations_ala_l2l
      train_loss, train_loss_std, train_acc, train_acc_std = meta_learner(task_dataset, call_backward=True)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/nn/modules/module.py&quot;, line 1051, in _call_impl
      return forward_call(*input, **kwargs)
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py&quot;, line 371, in forward
      meta_loss, meta_loss_std, meta_acc, meta_acc_std = forward(meta_learner=self,
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py&quot;, line 312, in forward
      loss, acc = fast_adapt(
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py&quot;, line 266, in fast_adapt
      learner.adapt(adaptation_error)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/algorithms/maml.py&quot;, line 169, in adapt
      self.module = maml_update(self.module, self.lr, gradients)
  UnboundLocalError: local variable 'gradients' referenced before assignment
  terminate called after throwing an instance of 'c10::CUDAError'
    what():  CUDA error: unspecified launch failure
  CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
  For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  Exception raised from query at ../aten/src/ATen/cuda/CUDAEvent.h:95 (most recent call first):
  frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f5036ae4a22 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libc10.so)
  frame #1: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x132 (0x7f50dbec70e2 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
  frame #2: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x50 (0x7f50dbec8d40 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
  frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x11c (0x7f50dbec975c in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
  frame #4: &lt;unknown function&gt; + 0xc71f (0x7f50da8aa71f in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
  frame #5: &lt;unknown function&gt; + 0x7ea5 (0x7f50e43e6ea5 in /lib64/libpthread.so.0)
  frame #6: clone + 0x6d (0x7f50e410fb0d in /lib64/libc.so.6)
  
  /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '
  ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 5 (pid: 158275) of binary: /home/miranda9/miniconda3/envs/meta_learning_a100/bin/python
  /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:367: UserWarning: 
  
  **********************************************************************
                 CHILD PROCESS FAILED WITH NO ERROR_FILE                
  **********************************************************************
  CHILD PROCESS FAILED WITH NO ERROR_FILE
  Child process 158275 (local_rank 5) FAILED (exitcode -6)
  Error msg: Signal 6 (SIGABRT) received by PID 158275
  Without writing an error file to &lt;N/A&gt;.
  While this DOES NOT affect the correctness of your application,
  no trace information about the error will be available for inspection.
  Consider decorating your top level entrypoint function with
  torch.distributed.elastic.multiprocessing.errors.record. Example:
  
    from torch.distributed.elastic.multiprocessing.errors import record
  
    @record
    def trainer_main(args):
       # do train
  **********************************************************************
    warnings.warn(_no_error_file_warning_msg(rank, failure))
  Traceback (most recent call last):
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/runpy.py&quot;, line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/runpy.py&quot;, line 87, in _run_code
      exec(code, run_globals)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py&quot;, line 702, in &lt;module&gt;
      main()
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py&quot;, line 361, in wrapper
      return f(*args, **kwargs)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py&quot;, line 698, in main
      run(args)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py&quot;, line 689, in run
      elastic_launch(
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launcher/api.py&quot;, line 116, in __call__
      return launch_agent(self._config, self._entrypoint, list(args))
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launcher/api.py&quot;, line 244, in launch_agent
      raise ChildFailedError(
  torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
  ****************************************************************************************************************************************
    /home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py FAILED  
  ========================================================================================================================================
  Root Cause:
  [0]:
    time: 2022-02-04_20:26:43
    rank: 5 (local_rank: 5)
    exitcode: -6 (pid: 158275)
    error_file: &lt;N/A&gt;
    msg: &quot;Signal 6 (SIGABRT) received by PID 158275&quot;
  ========================================================================================================================================
  Other Failures:
  [1]:
    time: 2022-02-04_20:26:43
    rank: 6 (local_rank: 6)
    exitcode: 1 (pid: 158280)
    error_file: &lt;N/A&gt;
    msg: &quot;Process failed with exitcode 1&quot;
  ****************************************************************************************************************************************
  </code></pre>
  <p>but different variations of the way to set up the script doesn't work.</p>
  <p>In summary, how is one suppose to use <code>torch.distributed.run</code> such that TWO or MORE scripts can run at once and they are all distributed?</p>
  <hr />
  <p>Related:</p>
  <ul>
  <li>main discussion for a solution here: <a href="https://github.com/learnables/learn2learn/issues/263" rel="nofollow noreferrer">https://github.com/learnables/learn2learn/issues/263</a></li>
  <li><a href="https://discuss.pytorch.org/t/multiprocessing-failed-with-torch-distributed-launch-module/33056" rel="nofollow noreferrer">https://discuss.pytorch.org/t/multiprocessing-failed-with-torch-distributed-launch-module/33056</a></li>
  <li>base code: <a href="https://github.com/learnables/learn2learn/blob/master/examples/vision/distributed_maml.py" rel="nofollow noreferrer">https://github.com/learnables/learn2learn/blob/master/examples/vision/distributed_maml.py</a></li>
  </ul>