title: Does PyTorch allocate GPU memory eagerly?
body: <p>Consider the following script:</p>
  <pre><code>import torch
  
  def unnecessary_compute():
      x = torch.randn(1000,1000, device='cuda')
      l = []
      for i in range(5):
          print(i,torch.cuda.memory_allocated())
          l.append(x**i)
  unnecessary_compute()
  </code></pre>
  <p>Running this script with PyTorch (1.11) generates the following output:</p>
  <pre><code>0 4000256
  1 8000512
  2 12000768
  3 16001024
  4 20971520
  </code></pre>
  <p>Given that <a href="https://pytorch.org/docs/master/notes/cuda.html#asynchronous-execution" rel="nofollow noreferrer">PyTorch uses asynchronous computation</a> and we never evaluated the contents of l or of a tensor that depends on l, why did PyTorch eagerly allocate GPU memory to the new tensors? Is there a way of invoking these tensors in an utterly lazy way (i.e., without triggering GPU memory allocation before it is required)?</p>