title: Why does setting backward(retain_graph=True) use up lot GPU memory?
body: <p>I need to backpropagate through my neural network multiple times, so I set <code>backward(retain_graph=True)</code>.</p>
  
  <p>However, this is causing </p>
  
  <blockquote>
    <p>RuntimeError: CUDA out of memory</p>
  </blockquote>
  
  <p>I don't understand why this is. </p>
  
  <p>Are the number of variables or weights doubling? Shouldn't the amount of memory used remain the same regardless of how many times <code>backward()</code> is called?</p>