title: tflearn / tensorflow does not learn xor
body: <p>Following code was written to learn the XOR function, but about half of the time the network does not learn and the loss after each epoch stays the same.</p>
  
  <pre><code>train_f = [[0, 0], [0, 1], [1, 0], [1, 1]]
  train_c = [[0], [1], [1], [0]]
  test_f = train_f
  test_c = train_c
  
  import tensorflow as tf
  import tflearn
  
  X = [[0., 0.], [0., 1.], [1., 0.], [1., 1.]]
  Y_xor = [[0.], [1.], [1.], [0.]]
  
  # Graph definition
  with tf.Graph().as_default():
      # Building a network with 2 optimizers
      net = tflearn.input_data(shape=[None, 2])
      # Nand operator definition
      net = tflearn.fully_connected(net, 2, activation='relu')
      net = tflearn.fully_connected(net, 2, activation='relu')
      net = tflearn.fully_connected(net, 1, activation='sigmoid')
      regressor = tflearn.regression(net, optimizer='adam', learning_rate=0.005, loss="mean_square",)
  
      # Training
      m = tflearn.DNN(regressor)
      m.fit(X, Y_xor, n_epoch=256, snapshot_epoch=False)
  
      # Testing
      print("Testing XOR operator")
      print("0 xor 0:", m.predict([[0., 0.]]))
      print("0 xor 1:", m.predict([[0., 1.]]))
      print("1 xor 0:", m.predict([[1., 0.]]))
      print("1 xor 1:", m.predict([[1., 1.]]))
  </code></pre>
  
  <p>Sometimes I get correct results like this:</p>
  
  <pre><code>Testing XOR operator
  0 xor 0: [[0.1487255096435547]]
  0 xor 1: [[0.9297153949737549]]
  1 xor 0: [[0.9354135394096375]]
  1 xor 1: [[0.1487255096435547]]
  </code></pre>
  
  <p>But often this:</p>
  
  <pre><code>Testing XOR operator
  0 xor 0: [[0.4999997615814209]]
  0 xor 1: [[0.5000002384185791]]
  1 xor 0: [[0.4999997615814209]]
  1 xor 1: [[0.5000001788139343]]
  </code></pre>
  
  <p>My 2x2x1 network should be able to perform XOR, and there is even some evidence that suggests that this network should always converge <a href="http://www.ncbi.nlm.nih.gov/pubmed/12662805">http://www.ncbi.nlm.nih.gov/pubmed/12662805</a> </p>
  
  <p>I have also tried to change the relu layers to sigmoid, to perform 2048 iterations, and to make a 4x4x1 and 6x6x1 networks, but the same problem still occurs sometimes.</p>
  
  <p>Could there be something wrong with how the weights are initialized?
  How do I use tflearn to have a neural net learn the xor function?</p>