title: Why does my Tensorflow based Gradient Descent breaks mid run?
body: <p>I am learning Tensorflow and am trying to make a shallow neural network on the MNIST dataset.</p>
  
  <p>The algorithm seems to be working until it reaches a certain point where it completely breaks. It goes back to randomly predicting the labels and does not improve.</p>
  
  <p>Thank you for your help.</p>
  
  <p>Code :</p>
  
  <pre class="lang-py prettyprint-override"><code># MNIST Data
  
  [train_data, cv_data, test_data] = loadMnistData()
  [x_train, y_train] = train_data
  [x_cv, y_cv] = cv_data
  [x_test, y_test] = test_data
  [m,n] = np.shape(x_train)
  [m,o] = np.shape(y_train)
  
  # Hyperparameters
  
  ALPHA = 0.5
  BATCH_SIZE = 10
  ITERATIONS = 30
  
  # Build
  
  X = tf.placeholder(tf.float32, [None, 784]) 
  y = tf.placeholder(tf.float32, [None, 10])
  
  W1 = tf.Variable(tf.random_normal([784, 30], stddev=.01))
  b1 = tf.Variable(tf.zeros([30]))
  
  W2 = tf.Variable(tf.random_normal([30, 10], stddev=.01))
  b2 = tf.Variable(tf.zeros([10]))
  
  # Forward Propagation
  
  z1 = tf.matmul(X, W1) + b1
  a1 = tf.sigmoid(z1)
  
  z2 = tf.matmul(a1, W2) + b2
  a2 = tf.sigmoid(z2)
  
  h = a2
  
  ce_cost = - tf.reduce_mean(tf.reduce_sum(y * tf.log(h) + (1 - y) * tf.log(1 - h), 1))
  
  accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(h, 1), tf.argmax(y, 1)), tf.float32))
  
  # Intermediate
  
  optimizer = tf.train.GradientDescentOptimizer(ALPHA)
  train = optimizer.minimize(ce_cost)
  
  sess = tf.InteractiveSession()
  tf.global_variables_initializer().run()
  
  for i in range(ITERATIONS):
  
    random_shuffling = np.arange(m)
    np.random.shuffle(random_shuffling)
  
    x_train = x_train[random_shuffling]
    y_train = y_train[random_shuffling]
  
    x_batches = [x_train[j : j + BATCH_SIZE] for j in range(0, m, BATCH_SIZE)]
    y_batches = [y_train[j : j + BATCH_SIZE] for j in range(0, m, BATCH_SIZE)]
  
    for x_batch, y_batch in zip(x_batches, y_batches):
  
      sess.run(train, {X : x_batch, y : y_batch})
  
    print(i, ": ", sess.run(accuracy, {X : x_test, y : y_test}))
  </code></pre>
  
  <p>Output :</p>
  
  <ul>
  <li>0 :  0.9399</li>
  <li>1 :  0.9456</li>
  <li>2 :  0.951</li>
  <li>3 :  0.9517</li>
  <li>4 :  0.9534</li>
  <li>5 :  0.9566</li>
  <li>6 :  0.9526</li>
  <li>7 :  0.098</li>
  <li>8 :  0.098</li>
  <li>...</li>
  <li>29 :  0.098</li>
  </ul>