title: Pinning memory is actually slower in PyTorch?
body: <p>I'm wondering why would pinning memory in PyTorch make things even slower. By reading the code of <code>torch.utils.data.dataloader</code>, I found the <code>pin_memory=True</code> option of <code>DataLoader</code> simply calls <code>.pin_memory()</code> on each batch before returning them. The returned tensor is still on CPU, and I have to call <code>.cuda(non_blocking=True)</code> manually after this. Therefore, the whole process would be</p>
  
  <pre class="lang-py prettyprint-override"><code>for x in some_iter:
      yield x.pin_memory().cuda(non_blocking=True)
  </code></pre>
  
  <p>I compared the performance of this with</p>
  
  <pre class="lang-py prettyprint-override"><code>for x in some_iter:
      yield x.cuda()
  </code></pre>
  
  <p>Here is the actual code</p>
  
  <pre class="lang-py prettyprint-override"><code>a = torch.rand(1024, 655360)
  
  %%time
  for i in a:
      i.pin_memory().cuda(non_blocking=True)
  # CPU times: user 1.35 s, sys: 55.8 ms, total: 1.41 s
  # Wall time: 396 ms
  
  %%time
  for i in a:
      i.pin_memory().cuda()
  # CPU times: user 1.6 s, sys: 12.2 ms, total: 1.62 s
  # Wall time: 404 ms
  
  %%time
  for i in a:
      i.cuda(non_blocking=True)
  # CPU times: user 855 ms, sys: 3.87 ms, total: 859 ms
  # Wall time: 274 ms
  
  %%time
  for i in a:
      i.cuda()
  # CPU times: user 314 ms, sys: 12 µs, total: 314 ms
  # Wall time: 313 ms
  </code></pre>
  
  <p>As a result, <strong>not pinning memory both uses less CPU time, and is faster in terms of actual time</strong>. Shouldn't pinning memory make data transfer asynchronous and therefore be faster? If that's not the case, why would we do pin memory?</p>
  
  <hr>
  
  <p>PS. I thought about the possibility of pinning a whole <code>TensorDataset</code> in advance (rather than pinning batches each time). But this cannot pin a tensor that is bigger than GPU memory:</p>
  
  <pre class="lang-py prettyprint-override"><code>a = np.memmap('../dat/R/train.3,31,31B', '3,31,31B', 'r')
  a.nbytes // 2**30
  ## 68
  
  torch.from_numpy(a).pin_memory()
  ## ---------------------------------------------------------------------------
  ## RuntimeError                              Traceback (most recent call last)
  ## &lt;ipython-input-36-d6f2d74da8e7&gt; in &lt;module&gt;
  ## ----&gt; 1 torch.from_numpy(a).pin_memory()
  ##
  ## RuntimeError: cuda runtime error (2) : out of memory at /tmp/pip-req-build-58y_cjjl/aten/src/THC/THCCachingHostAllocator.cpp:296
  </code></pre>
  
  <p>And if I do want to pin a small tensor, why don't I directly move the whole tensor into GPU memory in advance?</p>