title: MNIST Classification: mean_squared_error loss function and tanh activation function
body: <p>I changed the <a href="https://www.tensorflow.org/tutorials/" rel="nofollow noreferrer">getting started example</a> of Tensorflow as following:</p>
  
  <pre class="lang-py prettyprint-override"><code>import tensorflow as tf
  from sklearn.metrics import roc_auc_score
  import numpy as np
  import commons as cm
  from sklearn.metrics import confusion_matrix
  import matplotlib.pyplot as plt
  import pandas as pd
  import seaborn as sn
  
  mnist = tf.keras.datasets.mnist
  
  (x_train, y_train), (x_test, y_test) = mnist.load_data()
  x_train, x_test = x_train / 255.0, x_test / 255.0
  
  model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation=tf.nn.tanh),
    # tf.keras.layers.Dense(512, activation=tf.nn.tanh),
    # tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation=tf.nn.tanh)
  ])
  model.compile(optimizer='adam',
                 loss='mean_squared_error',
                # loss = 'sparse_categorical_crossentropy',
                metrics=['accuracy'])
  
  history = cm.Histories()
  h= model.fit(x_train, y_train, epochs=50, callbacks=[history])
  print("history:", history.losses)
  cm.plot_history(h)
  # cm.plot(history.losses, history.aucs)
  
  
  test_predictions = model.predict(x_test)
  
  
  # Compute confusion matrix
  pred = np.argmax(test_predictions,axis=1)
  pred2 = model.predict_classes(x_test)
  confusion = confusion_matrix(y_test, pred)
  cm.draw_confusion(confusion,range(10))
  </code></pre>
  
  <p>With its default parameters: </p>
  
  <ul>
  <li><code>relu</code> activation at hidden layers, </li>
  <li><code>softmax</code> at the output layer and </li>
  <li><code>sparse_categorical_crossentropy</code> as loss function, </li>
  </ul>
  
  <p>it works fine and the prediction for all digits are above 99%</p>
  
  <p>However with my parameters: <code>tanh</code> activation function and <code>mean_squared_error</code> loss function it just predict <code>0</code> for all test samples:</p>
  
  <p><a href="https://i.stack.imgur.com/CH7HG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CH7HG.png" alt="enter image description here"></a></p>
  
  <p>I wonder what is the problem? The accuracy rate is increasing for each epoch and it reaches 99% and loss is about 20</p>