title: Python - Low accuracy with low loss with tensorflow
body: <p>I'm building a simple neural network that takes 3 values and gives 2 outputs.</p>
  
  <p>I'm getting an accuracy of 67.5% and an average cost of 0.05</p>
  
  <p>I have a training dataset of 1000 examples and 500 testing examples. I plan on making a larger dataset in the near future.</p>
  
  <p>A little while ago I managed to get an accuracy of about 82% and sometimes a bit higher, but the cost was quite high.</p>
  
  <p>I've been experimenting with adding another layer which is currently in the model and that is the reason I have got the loss under 1.0</p>
  
  <p>I'm not sure what is going wrong, I'm new to Tensorflow and NNs in general.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  import sys
  sys.path.insert(0, '.../Dataset/Testing/')
  sys.path.insert(0, '.../Dataset/Training/')
  #other files
  from TestDataNormaliser import *
  from TrainDataNormaliser import *
  
  learning_rate = 0.01
  trainingIteration = 10
  batchSize = 100
  displayStep = 1
  
  
  x = tf.placeholder("float", [None, 3])
  y = tf.placeholder("float", [None, 2])
  
  
  
  #layer 1
  w1 = tf.Variable(tf.truncated_normal([3, 4], stddev=0.1))
  b1 = tf.Variable(tf.zeros([4])) 
  y1 = tf.matmul(x, w1) + b1
  
  #layer 2
  w2 = tf.Variable(tf.truncated_normal([4, 4], stddev=0.1))
  b2 = tf.Variable(tf.zeros([4]))
  #y2 = tf.nn.sigmoid(tf.matmul(y1, w2) + b2)
  y2 = tf.matmul(y1, w2) + b2
  
  w3 = tf.Variable(tf.truncated_normal([4, 2], stddev=0.1)) 
  b3 = tf.Variable(tf.zeros([2]))
  y3 = tf.nn.sigmoid(tf.matmul(y2, w3) + b3) #sigmoid
  
  
  #output
  #wO = tf.Variable(tf.truncated_normal([2, 2], stddev=0.1))
  #bO = tf.Variable(tf.zeros([2]))
  a = y3 #tf.nn.softmax(tf.matmul(y2, wO) + bO) #y2
  a_ = tf.placeholder("float", [None, 2])
  
  
  #cost function
  cross_entropy = tf.reduce_mean(-tf.reduce_sum(y * tf.log(a)))
  #cross_entropy = -tf.reduce_sum(y*tf.log(a))
  
  optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cross_entropy)
  
  
  #training
  
  init = tf.global_variables_initializer() #initialises tensorflow
  
  with tf.Session() as sess:
      sess.run(init) #runs the initialiser
  
      writer = tf.summary.FileWriter(".../Logs")
      writer.add_graph(sess.graph)
      merged_summary = tf.summary.merge_all()
  
      for iteration in range(trainingIteration):
          avg_cost = 0
          totalBatch = int(len(trainArrayValues)/batchSize) #1000/100
          #totalBatch = 10
  
          for i in range(batchSize):
              start = i
              end = i + batchSize #100
  
              xBatch = trainArrayValues[start:end]
              yBatch = trainArrayLabels[start:end]
  
              #feeding training data
  
              sess.run(optimizer, feed_dict={x: xBatch, y: yBatch})
  
              i += batchSize
  
              avg_cost += sess.run(cross_entropy, feed_dict={x: xBatch, y: yBatch})/totalBatch
  
              if iteration % displayStep == 0:
                  print("Iteration:", '%04d' % (iteration + 1), "cost=", "{:.9f}".format(avg_cost))
  
          #
      print("Training complete")
  
  
      predictions = tf.equal(tf.argmax(a, 1), tf.argmax(y, 1))
  
      accuracy = tf.reduce_mean(tf.cast(predictions, "float"))
      print("Accuracy:", accuracy.eval({x: testArrayValues, y: testArrayLabels}))
  </code></pre>