title: Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'
body: <p>I am using pytorch for training models. But I got an runtime error when it was computing the cross-entropy loss. </p>
  
  <pre><code>Traceback (most recent call last):
    File "deparser.py", line 402, in &lt;module&gt;
      d.train()
    File "deparser.py", line 331, in train
      total, correct, avgloss = self.train_util()
    File "deparser.py", line 362, in train_util
      loss = self.step(X_train, Y_train, correct, total)
    File "deparser.py", line 214, in step
      loss = nn.CrossEntropyLoss()(out.long(), y)
    File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
      result = self.forward(*input, **kwargs)
    File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 862, in forward
      ignore_index=self.ignore_index, reduction=self.reduction)
    File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
      return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
    File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 975, in log_softmax
      return input.log_softmax(dim)
  RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'
  </code></pre>
  
  <p>I think this is because the <code>.cuda()</code> function or conversion between <code>torch.Float</code> and <code>torch.Long</code>. But I have tried many ways to change the variable by <code>.cpu()</code>/<code>.cuda()</code> and <code>.long()</code>/<code>.float()</code>, but it still not work. This error message can't be found when searching it on google. Can anyone helps me? Thanks!!!</p>
  
  <p>This is the code cause error:</p>
  
  <pre><code>def step(self, x, y, correct, total):
      self.optimizer.zero_grad()
      out = self.forward(*x)
      loss = nn.CrossEntropyLoss()(out.long(), y)
      loss.backward()
      self.optimizer.step()
      _, predicted = torch.max(out.data, 1)
      total += y.size(0)
      correct += int((predicted == y).sum().data)
      return loss.data
  </code></pre>
  
  <p>And this function step() is called by: </p>
  
  <pre><code>def train_util(self):
      total = 0
      correct = 0
      avgloss = 0
      for i in range(self.step_num_per_epoch):
          X_train, Y_train = self.trainloader()
          self.optimizer.zero_grad()
          if torch.cuda.is_available():
              self.cuda()
              for i in range(len(X_train)):
                  X_train[i] = Variable(torch.from_numpy(X_train[i]))
                  X_train[i].requires_grad = False
                  X_train[i] = X_train[i].cuda()
              Y_train = torch.from_numpy(Y_train)
              Y_train.requires_grad = False
              Y_train = Y_train.cuda()
          loss = self.step(X_train, Y_train, correct, total)
          avgloss+=float(loss)*Y_train.size(0)
          self.optimizer.step()
          if i%100==99:
              print('STEP %d, Loss: %.4f, Acc: %.4f'%(i+1,loss,correct/total))
  
      return total, correct, avgloss/self.data_len
  </code></pre>
  
  <p>The input data <code>X_train, Y_train = self.trainloader()</code> are numpy arrays at begining.</p>
  
  <p>This is a data sample:</p>
  
  <pre><code>&gt;&gt;&gt; X_train, Y_train = d.trainloader()
  &gt;&gt;&gt; X_train[0].dtype
  dtype('int64')
  &gt;&gt;&gt; X_train[1].dtype
  dtype('int64')
  &gt;&gt;&gt; X_train[2].dtype
  dtype('int64')
  &gt;&gt;&gt; Y_train.dtype
  dtype('float32')
  &gt;&gt;&gt; X_train[0]
  array([[   0,    6,    0, ...,    0,    0,    0],
         [   0, 1944, 8168, ...,    0,    0,    0],
         [   0,  815,  317, ...,    0,    0,    0],
         ...,
         [   0,    0,    0, ...,    0,    0,    0],
         [   0,   23,    6, ...,    0,    0,    0],
         [   0,    0,  297, ...,    0,    0,    0]])
  &gt;&gt;&gt; X_train[1]
  array([ 6,  7,  8, 21,  2, 34,  3,  4, 19, 14, 15,  2, 13,  3, 11, 22,  4,
     13, 34, 10, 13,  3, 48, 18, 16, 19, 16, 17, 48,  3,  3, 13])
  &gt;&gt;&gt; X_train[2]
  array([ 4,  5,  8, 36,  2, 33,  5,  3, 17, 16, 11,  0,  9,  3, 10, 20,  1,
     14, 33, 25, 19,  1, 46, 17, 14, 24, 15, 15, 51,  2,  1, 14])
  &gt;&gt;&gt; Y_train
  array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
         [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         ...,
         [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
        dtype=float32)
  </code></pre>
  
  <p>Try all possible combinations:</p>
  
  <p>case 1:<br>
  <code>loss = nn.CrossEntropyLoss()(out, y)</code><br>
  I get:<br>
  <code>RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'</code>  </p>
  
  <p>case 2:<br>
  <code>loss = nn.CrossEntropyLoss()(out.long(), y)</code><br>
  as description above  </p>
  
  <p>case 3:<br>
  <code>loss = nn.CrossEntropyLoss()(out.float(), y)</code><br>
  I get:<br>
  <code>RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'</code></p>
  
  <p>case 4:<br>
  <code>loss = nn.CrossEntropyLoss()(out, y.long())</code><br>
  I get:<br>
  <code>RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15</code>  </p>
  
  <p>case 5:<br>
  <code>loss = nn.CrossEntropyLoss()(out.long(), y.long())</code><br>
  I get:<br>
  <code>RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'</code>  </p>
  
  <p>case 6:<br>
  <code>loss = nn.CrossEntropyLoss()(out.float(), y.long())</code><br>
  I get:<br>
  <code>RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15</code>  </p>
  
  <p>case 7:<br>
  <code>loss = nn.CrossEntropyLoss()(out, y.float())</code><br>
  I get:<br>
  <code>RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'</code>  </p>
  
  <p>case 8:<br>
  <code>loss = nn.CrossEntropyLoss()(out.long(), y.float())</code><br>
  I get:<br>
  <code>RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'</code>  </p>
  
  <p>case 9:<br>
  <code>loss = nn.CrossEntropyLoss()(out.float(), y.float())</code><br>
  I get:<br>
  <code>RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'</code>  </p>