title: TensorFlow: 2 layer feed forward neural net
body: <p>I'm trying to implement a simple fully-connected feed-forward neural net in TensorFlow (Python 3 version). The network has 2 inputs and 1 output, and I'm trying to train it to output the XOR of the two inputs. My code is as follows:</p>
  
  <pre><code>import numpy as np
  import tensorflow as tf
  
  sess = tf.InteractiveSession()
  
  inputs = tf.placeholder(tf.float32, shape = [None, 2])
  desired_outputs = tf.placeholder(tf.float32, shape = [None, 1])
  
  weights_1 = tf.Variable(tf.zeros([2, 3]))
  biases_1 = tf.Variable(tf.zeros([1, 3]))
  layer_1_outputs = tf.nn.sigmoid(tf.matmul(inputs, weights_1) + biases_1)
  
  weights_2 = tf.Variable(tf.zeros([3, 1]))
  biases_2 = tf.Variable(tf.zeros([1, 1]))
  layer_2_outputs = tf.nn.sigmoid(tf.matmul(layer_1_outputs, weights_2) + biases_2)
  
  error_function = -tf.reduce_sum(desired_outputs * tf.log(layer_2_outputs))
  train_step = tf.train.GradientDescentOptimizer(0.05).minimize(error_function)
  
  sess.run(tf.initialize_all_variables())
  
  training_inputs = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
  training_outputs = [[0.0], [1.0], [1.0], [0.0]]
  
  for i in range(10000):
      train_step.run(feed_dict = {inputs: np.array(training_inputs), desired_outputs: np.array(training_outputs)})
  
  print(sess.run(layer_2_outputs, feed_dict = {inputs: np.array([[0.0, 0.0]])}))
  print(sess.run(layer_2_outputs, feed_dict = {inputs: np.array([[0.0, 1.0]])}))
  print(sess.run(layer_2_outputs, feed_dict = {inputs: np.array([[1.0, 0.0]])}))
  print(sess.run(layer_2_outputs, feed_dict = {inputs: np.array([[1.0, 1.0]])}))
  </code></pre>
  
  <p>It seems simple enough, but the print statements at the end show that the neural net is nowhere near the desired outputs, regardless of number of training iterations or learning rate. Can anyone see what I am doing wrong?</p>
  
  <p>Thank you.</p>
  
  <p><strong>EDIT</strong>:
  I've also tried the following alternative error function:</p>
  
  <pre><code>error_function = 0.5 * tf.reduce_sum(tf.sub(layer_2_outputs, desired_outputs) * tf.sub(layer_2_outputs, desired_outputs))
  </code></pre>
  
  <p>That error function is the sum of the squares of the errors. It ALWAYS results in the network outputting a value of exactly 0.5-- another indication of a mistake somewhere in my code.</p>
  
  <p><strong>EDIT 2</strong>:
  I've found that my code works fine for AND and OR, but not for XOR. I'm extremely puzzled now.</p>