title: Why deep NN can't approximate simple ln(x) function?
body: <p>I have created ANN with two RELU hidden layers + linear activation layer and trying to approximate simple ln(x) function. And I am can't do this good. I am confused because lx(x) in x:[0.0-1.0] range should be approximated without problems (I am using learning rate 0.01 and basic grad descent optimization).</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  
  def GetTargetResult(x):
      curY = np.log(x)
      return curY
  
  # Create model
  def multilayer_perceptron(x, weights, biases):
      # Hidden layer with RELU activation
      layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
      layer_1 = tf.nn.relu(layer_1)
      # # Hidden layer with RELU activation
      layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
      layer_2 = tf.nn.relu(layer_2)
  
      # Output layer with linear activation
      out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
      return out_layer
  
  # Parameters
  learning_rate = 0.01
  training_epochs = 10000
  batch_size = 50
  display_step = 500
  
  # Network Parameters
  n_hidden_1 = 50 # 1st layer number of features
  n_hidden_2 = 10 # 2nd layer number of features
  n_input =  1
  
  
  # Store layers weight &amp; bias
  weights = {
      'h1': tf.Variable(tf.random_uniform([n_input, n_hidden_1])),
      'h2': tf.Variable(tf.random_uniform([n_hidden_1, n_hidden_2])),
      'out': tf.Variable(tf.random_uniform([n_hidden_2, 1]))
  }
  biases = {
      'b1': tf.Variable(tf.random_uniform([n_hidden_1])),
      'b2': tf.Variable(tf.random_uniform([n_hidden_2])),
      'out': tf.Variable(tf.random_uniform([1]))
  }
  
  x_data = tf.placeholder(tf.float32, [None, 1])
  y_data = tf.placeholder(tf.float32, [None, 1])
  
  # Construct model
  pred = multilayer_perceptron(x_data, weights, biases)
  
  # Minimize the mean squared errors.
  loss = tf.reduce_mean(tf.square(pred - y_data))
  optimizer = tf.train.GradientDescentOptimizer(learning_rate)
  train = optimizer.minimize(loss)
  
  # Before starting, initialize the variables.  We will 'run' this first.
  init = tf.initialize_all_variables ()
  
  # Launch the graph.
  sess = tf.Session()
  sess.run(init)
  
  for step in range(training_epochs):
      x_in = np.random.rand(batch_size, 1).astype(np.float32)
      y_in = GetTargetResult(x_in)
      sess.run(train, feed_dict = {x_data: x_in, y_data: y_in})
      if(step % display_step == 0):
          curX = np.random.rand(1, 1).astype(np.float32)
          curY =  GetTargetResult(curX)
  
          curPrediction = sess.run(pred, feed_dict={x_data: curX})
          curLoss = sess.run(loss, feed_dict={x_data: curX, y_data: curY})
          print("For x = {0} and target y = {1} prediction was y = {2} and squared loss was = {3}".format(curX, curY,curPrediction, curLoss))
  </code></pre>
  
  <p>For the configuration above NN is just learning to guess y = -1.00. I have tried different learning rates,  couple optimizers and different configurations with no success - learning does not converge in any case. I did something like that with logarithm in past in other deep learning framework without problem.. Can be the TF specific issue? What am I doing wrong?</p>