title: Tensorflow model not training when changing the input data
body: <p>I am trying to use tensorflow to train a neural network (LeNet) using the traffic sign images.  I want to check the effect of a preprocessing technique on the performance of the nn. So, I preprocessed the images and stored the results (trainingimages, validationimages,testimages, final testimages) as a tuple in a dict.</p>
  
  <p>I then tried to iterate over this dict and then use the training and validation operations of the tensorflow as follows</p>
  
  <pre><code>import tensorflow as tf
  from sklearn.utils import shuffle
  
  
  output_data = []
  EPOCHS = 5
  BATCH_SIZE = 128
  rate = 0.0005
  
  for key in finalInputdata.keys():
      for procTypes in range(0,(len(finalInputdata[key]))):
          if np.shape(finalInputdata[key][procTypes][0]) != ():
              X_train = finalInputdata[key][procTypes][0]
              X_valid = finalInputdata[key][procTypes][1]
              X_test = finalInputdata[key][procTypes][2]
              X_finaltest = finalInputdata[key][procTypes][3]
  
  
              x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1]))
              y = tf.placeholder(tf.int32, (None))
              one_hot_y = tf.one_hot(y,43)
  
              # Tensor Operations
              logits = LeNet(x,np.shape(X_train)[-1])
  
              cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,one_hot_y)
              softmax_probability = tf.nn.softmax(logits)
  
              loss_operation = tf.reduce_mean(cross_entropy)
              optimizer = tf.train.AdamOptimizer(learning_rate=rate)
              training_operation = optimizer.minimize(loss_operation)
              correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(one_hot_y,1))
              accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
  
  
                  # Pipeline for training and evaluation
  
              sess = tf.InteractiveSession()
  
              sess.run(tf.global_variables_initializer())
              num_examples = len(X_train)
  
              print("Training on %s images processed as %s" %(key,dict_fornames['proctypes'][procTypes]))
              print()
              for i in range(EPOCHS):
                  X_train, y_train = shuffle(X_train, y_train)
                  for offset in range(0, num_examples, BATCH_SIZE):
                      end = offset + BATCH_SIZE
                      batch_x, batch_y = X_train[offset:end], y_train[offset:end]
                      sess.run(training_operation, feed_dict = {x: batch_x, y: batch_y})
  
                  training_accuracy = evaluate(X_train,y_train)
  
                  validation_accuracy = evaluate(X_valid, y_valid)
  
                  testing_accuracy = evaluate(X_test, y_test)
  
                  final_accuracy = evaluate(X_finaltest, y_finalTest)
  
                  print("EPOCH {} ...".format(i+1))
                  print("Training Accuracy = {:.3f}".format(training_accuracy))
                  print("Validation Accuracy = {:.3f}".format(validation_accuracy))
                  print()
                  output_data.append({'EPOCHS':EPOCHS, 'LearningRate':rate, 'ImageType': 'RGB',\
                                      'PreprocType': dict_fornames['proctypes'][0],\
                                      'TrainingAccuracy':training_accuracy, 'ValidationAccuracy':validation_accuracy, \
                                      'TestingAccuracy': testing_accuracy})
  
  
              sess.close()
  </code></pre>
  
  <p>The evaluate function is as follows</p>
  
  <pre><code>def evaluate(X_data, y_data):
      num_examples = len(X_data)
      total_accuracy = 0
      sess = tf.get_default_session()
      for offset in range(0,num_examples, BATCH_SIZE):
          batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
          accuracy = sess.run(accuracy_operation, feed_dict = {x:batch_x, y:batch_y})
          total_accuracy += (accuracy * len(batch_x))
      return total_accuracy / num_examples
  </code></pre>
  
  <p>Once I execute the program, it works good for the first iteration of dataset but from the second iteration, the network doesnt train and continues to do so for all the other iterations. </p>
  
  <pre><code>Training on RGB images processed as Original
  
  EPOCH 1 ...
  Training Accuracy = 0.525
  Validation Accuracy = 0.474
  
  EPOCH 2 ...
  Training Accuracy = 0.763
  Validation Accuracy = 0.682
  
  EPOCH 3 ...
  Training Accuracy = 0.844
  Validation Accuracy = 0.723
  
  EPOCH 4 ...
  Training Accuracy = 0.888
  Validation Accuracy = 0.779
  
  EPOCH 5 ...
  Training Accuracy = 0.913
  Validation Accuracy = 0.795
  
  Training on RGB images processed as Mean Subtracted Data
  
  EPOCH 1 ...
  Training Accuracy = 0.056
  Validation Accuracy = 0.057
  
  EPOCH 2 ...
  Training Accuracy = 0.057
  Validation Accuracy = 0.057
  
  EPOCH 3 ...
  Training Accuracy = 0.057
  Validation Accuracy = 0.056
  
  EPOCH 4 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.056
  
  EPOCH 5 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.058
  
  Training on RGB images processed as Normalized Data
  
  EPOCH 1 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.054
  
  EPOCH 2 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.054
  
  EPOCH 3 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.054
  
  EPOCH 4 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.054
  
  EPOCH 5 ...
  Training Accuracy = 0.058
  Validation Accuracy = 0.054
  </code></pre>
  
  <p>However, if I restart the kernel and use any datatype (any iteration), it works. I figured out that I must clear the graph or run multiple sessions for multiple datatypes but I am not yet clear on how to do that. I tried using <code>tf.reset_default_graph()</code> but seems like it does not have any effect. Can somebody point me in the right direction?</p>
  
  <p>Thanks</p>