title: Should I use softmax as output when using cross entropy loss in pytorch?
body: <p>I have a problem with classifying fully connected deep neural net with 2 hidden layers for <strong>MNIST dataset in pytorch</strong>.</p>
  <p>I want to use <strong>tanh</strong> as activations in both hidden layers, but in the end, I should use <strong>softmax</strong>.</p>
  <p>For the loss, I am choosing <code>nn.CrossEntropyLoss()</code> in PyTOrch, which (as I have found out) does not want to take one-hot encoded labels as true labels, but takes LongTensor of classes instead.</p>
  <p>My model is <code>nn.Sequential()</code> and when I am using softmax in the end, it gives me worse results in terms of accuracy on testing data. Why?</p>
  <pre class="lang-py prettyprint-override"><code>import torch
  from torch import nn
  
  inputs, n_hidden0, n_hidden1, out = 784, 128, 64, 10
  n_epochs = 500
  model = nn.Sequential(
      nn.Linear(inputs, n_hidden0, bias=True), 
      nn.Tanh(),
      nn.Linear(n_hidden0, n_hidden1, bias=True),
      nn.Tanh(),
      nn.Linear(n_hidden1, out, bias=True),
      nn.Softmax()  # SHOULD THIS BE THERE?
  )
                   
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.5)
  
  for epoch in range(n_epochs):
      y_pred = model(X_train)
      loss = criterion(y_pred, Y_train)
      print('epoch: ', epoch+1,' loss: ', loss.item())
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
  </code></pre>