title: When does one have to call share_memory_() in Pytorch when using distributed training?
body: <p>I want to train things in parallel/distributed in 1 machine using multiple CPUS or GPUs. Thus my question is when does one ever run:</p>
  <pre><code>        # I DONT THINK THIS IS NEEDED, since you don't need to share data...each process loads in own stuff and uses it
          x, y = x.share_memory_(), y.share_memory_()
  </code></pre>
  <p>as my comment says, my belief is that <code>.share_memory_()</code> is never actually needed in pytorch because:</p>
  <ol>
  <li>when using gpu you have to move the data to the correct GPU anyway so you would do <code>x.to(rank)</code>.</li>
  <li>when using cpus, each process would have a copy of their own data (sending data is expensive in distributed training as far as I understand), so there is no need to have it &quot;shared&quot; with <code>.shared_memory_()</code> since they are already reading the same data from disk (or something similar to that I assume).</li>
  </ol>
  <p>Are these assumptions correct? Is it true that for data we never need <code>.memory_share_()</code>? Btw <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.share_memory_" rel="nofollow noreferrer">the docs say</a>:</p>
  <blockquote>
  <p>Moves the underlying storage to shared memory.</p>
  </blockquote>
  <blockquote>
  <p>This is a no-op if the underlying storage is already in shared memory and for CUDA tensors. Tensors in shared memory cannot be resized.</p>
  </blockquote>
  <p>Note however that if one is using CPUs only for the model I think we do need:</p>
  <pre><code>    else:
      # if we want multiple cpu just make sure the model is shared properly accross the cpus with shared_memory()
      # note that op is a no op if it's already in shared_memory
          model = model.share_memory()
          ddp_model = DDP(model)  # I think removing the devices ids should be fine...?
  </code></pre>
  <hr />
  <p>related:</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/66226135/how-to-parallelize-a-training-loop-ever-samples-of-a-batch-when-cpu-is-only-avai">How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch?</a></li>
  </ul>