title: Why can't I fit this simple tensorflow model?
body: <p>I have a very simple tensorflow model.</p>
  
  <p>However, I can't seem to train the model to give the desired output.</p>
  
  <p>Example code is below:</p>
  
  <pre><code>import numpy as np
  import tensorflow as tf
  
  input_values = np.array([
      [[0.3], [0.3], [0.2]],  # Sum 0.8
      [[0.2], [0.3], [0.4]],  # Sum 0.9
      [[0.9], [0.05], [0.05]],  # Sum 1.0
      [[0.1], [0.1], [0.9]],  # Sum 1.1
      [[0.2], [0.5], [0.5]],  # Sum 1.2
  ])
  
  target_values = np.array([
      [0.8], [0.9], [1.0], [1.1], [1.2]
  ])
  
  targets = tf.Variable(target_values, trainable=False)
  
  inputs = tf.placeholder(tf.float32, shape=(None, None, 1))
  
  w0 = tf.get_variable('w0', shape=(1, 8), initializer=tf.truncated_normal_initializer())
  b0 = tf.get_variable('b0', shape=8, initializer=tf.truncated_normal_initializer())
  
  w1 = tf.get_variable('w1', shape=(8, 1), initializer=tf.truncated_normal_initializer())
  b1 = tf.get_variable('b1', shape=1, initializer=tf.truncated_normal_initializer())
  
  output = tf.reshape(inputs, (-1, 1))
  output = tf.nn.relu(tf.matmul(output, w0) + b0)
  output = tf.nn.relu(tf.matmul(output, w1) + b1)
  output = tf.reshape(output, (5, 3, 1))
  output = tf.reduce_sum(output, axis=1)
  
  loss = tf.losses.mean_squared_error(output, targets)
  
  feed_dict = {inputs: input_values}
  
  sess = tf.InteractiveSession()
  sess.run(tf.global_variables_initializer())
  
  train_step = tf.train.GradientDescentOptimizer(0.001).minimize(loss)
  
  for _ in range(1000):
      sess.run(train_step, feed_dict=feed_dict)
  
  print('final loss : %0.2f' % loss.eval(feed_dict))
  print('output : \n%s' % output.eval(feed_dict))
  </code></pre>
  
  <p>However, after 1,000 training iterations, it comes out as something like:</p>
  
  <pre><code>[array([[ 1.16963911],
         [ 1.15914857],
         [ 0.75045645],
         [ 0.75721866],
         [ 0.97209203]], dtype=float32)]
  </code></pre>
  
  <p>But my desired output is:</p>
  
  <pre><code>[array([[ 0.8],
         [ 0.9],
         [ 1.0],
         [ 1.1],
         [ 1.2]], dtype=float32)]
  </code></pre>
  
  <p>I would have hoped that such a simple model would fit very precisely and in a small number of training iterations.</p>
  
  <p>The model is flexible enough to match the desired output. Setting the weights to these values will give the desired output:</p>
  
  <pre><code>w0 = tf.Variable([[1, 0, 0, 0, 0, 0, 0, 0]], dtype=tf.float32)
  b0 = tf.get_variable('b0', shape=8, initializer=tf.zeros_initializer())
  
  w1 = tf.Variable([[1], [0], [0], [0], [0], [0], [0], [0]], dtype=tf.float32)
  b1 = tf.get_variable('b1', shape=1, initializer=tf.zeros_initializer())
  </code></pre>
  
  <p>How can I:</p>
  
  <ul>
  <li>fit the model more accurately.</li>
  <li>fit the model in as few iterations as possible?</li>
  </ul>
  
  <p><strong>Update:</strong></p>
  
  <p>If I change the activation function from tf.nn.relu to tf.nn.sigmoid it can make a reasonable fit after 100,000 iterations (59 seconds on my machine).</p>
  
  <p>Here is the output:</p>
  
  <pre><code>[[ 0.8082366 ]
   [ 0.906322  ]
   [ 1.04254854]
   [ 1.09336698]
   [ 1.21020865]]
  </code></pre>
  
  <ul>
  <li>Why would a sigmoid function perform better in this scenario?</li>
  <li>Why does a sigmoid seem to train faster than the relu function in this scenario?</li>
  </ul>