title: "CUDA out of memory" in PyTorch
body: <p>I have
  16 GB of RAM
  AND GPU 2060 6 GB</p>
  <p>trying to run the <a href="https://huggingface.co/gpt2" rel="nofollow noreferrer">&quot;transformers gpt2 model&quot;</a> on GPU</p>
  <p>when I run  the code with CPU its work but take a long time (2 - 4) min
  so I tried to move to GPU but there is an &quot;out of memory&quot; issue</p>
  <pre><code>import torch
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  
  from transformers import GPT2LMHeadModel , GPT2Tokenizer
   
  #import torch
  
  print(device)
   
  
  
  # %% [markdown]
  # ### Load Model
  
  # %%
  tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
  
   
  
  model = GPT2LMHeadModel.from_pretrained('gpt2-large' , pad_token_id = tokenizer.eos_token_id )
  
   
  model.to(device) ## model to GPU
  
  
  # %%
  tokenizer.decode(tokenizer.eos_token_id)
  
  # %% [markdown]
  # ### Tokenize the text
  
  # %%
  sentence = &quot;workd war 2&quot;
  input_ids = tokenizer.encode(sentence , return_tensors = 'pt')
  
  # %%
  input_ids = input_ids.to(device)
  #prompt = prompt.to(&quot;cuda&quot;)
  
  
  # %%
  tokenizer.decode(input_ids[0])
  
  torch.cuda.empty_cache()
  
  # %%
  print(tokenizer.decode(input_ids[0][1]))
  #print(tokenizer.decode(input_ids[0][2]))
  #print(tokenizer.decode(input_ids[0][3]))
  #print(tokenizer.decode(input_ids[0][4]))
  
  # %% [markdown]
  # ### Generate and Decode Text
  
  # %%
  output = model.generate(input_ids, max_length = 500, num_beams = 5,no_repeat_ngram_size  = 2 , early_stopping = True) 
   
   
  # %%
  output[0]
  
  
  # %%
  print(tokenizer.decode(output[0] , skip_special_tokens = True))
  
  # %% [markdown]
  # ### Output Result
  
  # %%
  text = tokenizer.decode(output[0] , skip_special_tokens = True)
  
  
  # %%
  with open('okokok.txt','w') as f:
      f.write(text)
  </code></pre>
  <p>the error message I got</p>
  <pre><code>\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py&quot;, line 235, in forward
      present = torch.stack((key.transpose(-2, -1), value))  # transpose to have same shapes for stacking
  RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 6.00 GiB total capacity; 3.91 GiB already allocated; 0 bytes free; 4.57 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  </code></pre>
  <p>max memory used is 9 GB when running the code is that because of GPU memory or RAM memory?
  It must use the GPU for processing</p>