title: PyTorch Input and hidden tensors not on the same device
body: <p>I'm creating a simple LSTM model to predict some sales data. I am trying to train it on a GPU, but there seems to be a problem with casting the hidden state tensor to cuda.
  I get the following error:
  <code>RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu</code>.
  How can I train the model on a GPU? I cast the training data, initial hidden states, and the model to cuda, yet I still get the error.</p>
  <p>Here's my code:</p>
  <pre><code># Convert train_norm from an array to a tensor
  train_norm = torch.FloatTensor(train_norm).view(-1).cuda()
  
  # define a window size
  window_size = 12
  
  # Define function to create seq/label tuples
  
  def input_data(seq, ws): # ws is window size
      out = []
      L = len(seq)
  
      for i in range(L-ws):
          window = seq[i:i+ws]
          label = seq[i+ws:i+ws+1]
  
          out.append((window, label))
  
      return out
  
  # Apply the input_data function to the train_norm
  train_data = input_data(train_norm, window_size)
  
  class LSTM(nn.Module):
      def __init__(self, input_size=1, hidden_size=100, output_size=1):
          super().__init__()
          self.hidden_size = hidden_size
          
          # Add an LSTM layer:
          self.lstm = nn.LSTM(input_size, hidden_size)
  
          # Add a fully connected linear layer:
          self.linear = nn.Linear(hidden_size, output_size)
  
          # Initialize h0 and c0:
          self.hidden = (torch.zeros(1, 1, hidden_size).cuda(), torch.zeros(1, 1, hidden_size).cuda())
  
      def forward(self, seq):
          lstm_out, self.hidden = self.lstm(seq.view(len(seq), 1, -1), self.hidden)
          pred = self.linear(lstm_out.view(len(seq), -1))
  
          return pred[-1] # get only the last value
  
  model = LSTM().cuda()
  
  criterion = nn.MSELoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
  
  epochs = 200
  
  import time
  start_time = time.time()
  
  for epoch in range(epochs):
      
      # Extract the sequence and label from the training data
      for seq, y_train in train_data:
  
          # Reset the parameters and hidden states
          optimizer.zero_grad()
          hidden = (torch.zeros(1, 1, model.hidden_size),
                    torch.zeros(1, 1, model.hidden_size))
          model.hidden = hidden
  
          # Predict the values
          y_pred = model(seq)
  
          # Calculate loss and perform backpropagation
          loss = criterion(y_pred, y_train)
          loss.backward()
          optimizer.step()
  
      print(f'epoch: {epoch+1:2} loss: {loss.item():10.8f}')
  
  print(f'Training took {time.time() - start_time:.0f} seconds')
  </code></pre>