title: tf.keras predictions are bad while evaluation is good
body: <p>I'm programming a model in tf.keras, and running model.evaluate() on the training set usually yields ~96% accuracy. My evaluation on the test set is usually close, about 93%. However, when I predict manually, the model is usually inaccurate. This is my code:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>import tensorflow as tf
  from tensorflow.keras import layers
  import numpy as np
  import pandas as pd
  
  !git clone https://github.com/DanorRon/data
  %cd data
  !ls
  
  batch_size = 100
  epochs = 15
  alpha = 0.001
  lambda_ = 0.001
  h1 = 50
  
  train = pd.read_csv('/content/data/mnist_train.csv.zip')
  test = pd.read_csv('/content/data/mnist_test.csv.zip')
  
  train = train.loc['1':'5000', :]
  test = test.loc['1':'2000', :]
  
  train = train.sample(frac=1).reset_index(drop=True)
  test = test.sample(frac=1).reset_index(drop=True)
  
  x_train = train.loc[:, '1x1':'28x28']
  y_train = train.loc[:, 'label']
  
  x_test = test.loc[:, '1x1':'28x28']
  y_test = test.loc[:, 'label']
  
  x_train = x_train.values
  y_train = y_train.values
  
  x_test = x_test.values
  y_test = y_test.values
  
  nb_classes = 10
  targets = y_train.reshape(-1)
  y_train_onehot = np.eye(nb_classes)[targets]
  
  nb_classes = 10
  targets = y_test.reshape(-1)
  y_test_onehot = np.eye(nb_classes)[targets]
  
  model = tf.keras.Sequential()
  model.add(layers.Dense(784, input_shape=(784,), kernel_initializer='random_uniform', bias_initializer='zeros'))
  model.add(layers.Dense(h1, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(lambda_), kernel_initializer='random_uniform', bias_initializer='zeros'))
  model.add(layers.Dense(10, activation='softmax', kernel_regularizer=tf.keras.regularizers.l2(lambda_), kernel_initializer='random_uniform', bias_initializer='zeros'))
  
  model.compile(optimizer='SGD',
               loss = 'mse',
               metrics = ['categorical_accuracy'])
  
  model.fit(x_train, y_train_onehot, epochs=epochs, batch_size=batch_size)
  
  model.evaluate(x_test, y_test_onehot, batch_size=batch_size)
  
  prediction = model.predict_classes(x_test)
  print(prediction)
  
  print(y_test[1:])
  </code></pre>
  
  <p>I've heard that a lot of the time when people have this problem, it's just a problem with data input. But I can't see any problem with that here since it almost always predicts wrongly (about as much as you would expect if it was random). How do I fix this problem?</p>
  
  <p>Edit: Here are the specific results:</p>
  
  <p>Last training step:</p>
  
  <pre class="lang-python prettyprint-override"><code>Epoch 15/15
  49999/49999 [==============================] - 3s 70us/sample - loss: 0.0309 - categorical_accuracy: 0.9615
  </code></pre>
  
  <p>Evaluation output:</p>
  
  <pre class="lang-python prettyprint-override"><code>2000/2000 [==============================] - 0s 54us/sample - loss: 0.0352 - categorical_accuracy: 0.9310
  [0.03524150168523192, 0.931]
  </code></pre>
  
  <p>Output from model.predict_classes:</p>
  
  <pre class="lang-python prettyprint-override"><code>[9 9 0 ... 5 0 5]
  </code></pre>
  
  <p>Output from print(y_test):</p>
  
  <pre class="lang-python prettyprint-override"><code>[9 0 0 7 6 8 5 1 3 2 4 1 4 5 8 4 9 2 4]
  </code></pre>