title: How to reproduce Keras SimpleRNN behaviour
body: <p>I'm trying to learn how <code>keras.layers.SimpleRNN</code> works by following a relatively straightforward tutorial (<a href="https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/" rel="nofollow noreferrer">https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/</a>). However, this tutorial assumes the input is scalar, and I've been unable to scale this to higher-dimensional inputs. Here is my attempt to reproduce the behaviour of a simple RNN with 1x2 input for 3 time steps:</p>
  <pre><code>import keras
  
  inputs = np.array([[[1, 2]]])
  inputs = np.repeat(inputs, repeats = 3, axis=1)
  inputs.shape
  
  model = keras.Sequential()
  model.add(keras.Input(shape=(3, 2), name='input'))
  model.add(layers.SimpleRNN(4, name='rnn', activation='linear'))
  model.add(layers.Dense(5, name='output', activation='linear'))
  
  outputs = model(inputs)
  
  w_inputs = model.get_weights()[0]
  w_hidden = model.get_weights()[1]
  b_hidden = model.get_weights()[2]
  w_dense = model.get_weights()[3]
  b_dense = model.get_weights()[4]
  
  h0 = np.zeros(4)
  h1 = np.matmul(np.array([[1, 2]]), w_inputs) + h0 + b_hidden
  h2 = np.matmul(np.array([[1, 2]]), w_inputs) + b_hidden
  h3 = np.matmul(np.array([[1, 2]]), w_inputs) + b_hidden
  o3 = np.matmul(h3, w_dense) + b_dense
  
  print(f'output: {outputs}')
  print(f'expected: {o3}')
  </code></pre>
  <p>The output generated from <code>keras</code> and the values I calculated myself don't match:</p>
  <pre><code>output: [[-6.937807    4.0890574  -0.15574443 -1.2262737   1.9948364 ]]
  expected: [[-2.38697288  1.03174649 -1.44180991 -1.28951854  0.39630487]]
  </code></pre>
  <p>why?</p>