title: Why does this RNN in tensorflow not learn?
body: <p>I am trying to train an RNN without using the RNN API in  tensorflow (2) in Python 3.7, so the code is very basic. Something is going really wrong, but I'm not sure what it is.</p>
  
  <p>As a reference, I am using a dataset from this  <a href="https://www.tensorflow.org/tutorials/structured_data/time_series" rel="nofollow noreferrer">tensorflow tutorial</a> so I know what the error should roughly converge to. My RNN code is the following. What it is trying to do is use the previous 20 timesteps to predict the value of a series at the 21st timestep. I am training in batches of size 256.</p>
  
  <p>While there is a decrease in loss over time, the ceiling is approximately 10x what it is if I follow the tutorial approach. Could it be some problem with the backpropagation through time?</p>
  
  <pre><code>state_size = 20  #dimensionality of the network
  BATCH_SIZE = 256
  #define recurrent weights and biases. W has 1 more dimension that the state 
  #dimension as also processes the inputs
  W = tf.Variable(np.random.rand(state_size+1, state_size), dtype=tf.float32)
  b = tf.Variable(np.zeros((1,state_size)), dtype=tf.float32)
  
  #weights and biases for the output
  W2 = tf.Variable(np.random.rand(state_size, 1),dtype=tf.float32)
  b2 = tf.Variable(np.zeros((1,1)), dtype=tf.float32)
  
  
  init_state = tf.Variable(np.random.normal(size=[BATCH_SIZE,state_size]),dtype='float32')
  optimizer = tf.keras.optimizers.Adam(1e-3)
  losses = []
  for epoch in range(20):
      with tf.GradientTape() as tape:
          loss = 0
  
          for batch_idx in range(200):
  
              current_state = init_state
  
              batchx = x_train_uni[batch_idx*BATCH_SIZE:(batch_idx+1)*BATCH_SIZE].swapaxes(0,1)
              batchy = y_train_uni[batch_idx*BATCH_SIZE:(batch_idx+1)*BATCH_SIZE]
              #forward pass through the timesteps
              for x in batchx:
                  inst = tf.concat([current_state,x],1)  #concatenate state and inputs for that timepoint
                  current_state = tf.tanh(tf.matmul(inst, W) + b)  #
              #predict using the hidden state after the full forward pass
              pred = tf.matmul(current_state,W2) + b2
  
              loss += tf.reduce_mean(tf.abs(batchy-pred))
  
      #get gradients with respect to parameters
      gradients = tape.gradient(loss, [W,b,W2,b2])
      #apply gradients
      optimizer.apply_gradients(zip(gradients, [W,b,W2,b2]))
      losses.append(loss)
      print(loss)
  </code></pre>