title: Unable to learn a second inflection in this simple Tensorflow nnet?
body: <p>As a toy example I'm trying to fit a single hidden layer neural network to a sine wave. However, I only seem to be able to get the first inflection of the sine wave, after which it levels off and becomes flat:
  <a href="https://i.stack.imgur.com/9E8zd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9E8zd.png" alt="nnet output"></a>
  I've tried varying the amount of data, learning rate, hidden layer size, etc. but I always get something akin to this. Seems to me that something is fundamentally wrong in my code... From playing with this brilliant app this should be possible with 5-6 hidden units, although I've noticed that with certain weight initialisations it doesn't converge here either: <a href="http://neuron.eng.wayne.edu/bpFunctionApprox/bpFunctionApprox.html" rel="nofollow noreferrer">http://neuron.eng.wayne.edu/bpFunctionApprox/bpFunctionApprox.html</a></p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  sess = tf.InteractiveSession()
  
  hidden_units = 6 # 6 is theoretically a minimum for this function
  epochs = 50
  minibatch = 1
  
  def shuffle_in_unison(a, b):
      rng_state = np.random.get_state()
      np.random.shuffle(a)
      np.random.set_state(rng_state)
      np.random.shuffle(b)
  
  x_data = np.arange(0, 4 *np.pi, 0.3)
  y_data = np.sin(x_data + np.pi)
  
  x = tf.placeholder(tf.float32, shape=[None, 1])
  y_ = tf.placeholder(tf.float32, shape=[None, 1])
  
  W = tf.Variable(tf.random_normal([1, hidden_units], stddev=0.01), name="W")
  b = tf.Variable(tf.zeros([hidden_units]), name="b")
  W2 = tf.Variable(tf.random_normal([hidden_units, 1], stddev=0.01), name="W2")
  b2 = tf.Variable(tf.zeros([1]), name="b2")
  
  h = tf.nn.tanh(tf.matmul(x, W) + b)
  y = tf.matmul(h, W2) + b2
  
  squared_error = tf.reduce_mean(tf.square(y_ - y)) # tf.abs similar result
  train_step = tf.train.GradientDescentOptimizer(0.1).minimize(squared_error)
  
  sess.run(tf.initialize_all_variables())
  
  iterations = len(x_data) / minibatch
  total_iterations = iterations * epochs
  
  for epoch in range(epochs):
      tmp_x = x_data.copy()
      tmp_y = y_data.copy()
      shuffle_in_unison(tmp_x, tmp_y)
      for i in range(iterations):
          start_i = i*minibatch
          end_i   = (start_i + minibatch) % len(x_data) 
          x_batch = np.transpose(np.array([tmp_x[start_i:end_i]]))
          y_batch = np.transpose(np.array([tmp_y[start_i:end_i]]))
          train_step.run(feed_dict={x: x_batch, y_: y_batch})
  
  y_test = [y.eval(feed_dict={x:np.transpose(np.array([[a]]))})[0][0] for a in x_data]
  
  import matplotlib.pyplot as plt
  plt.plot(x_data, y_data)
  plt.plot(x_data, y_test)
  plt.legend(['Train', 'test'])
  plt.show()
  </code></pre>