<p>To follow up on @lemm-ras's <a href="https://stackoverflow.com/a/38907274/957657">answer</a>, <a href="https://arxiv.org/abs/1511.03771" rel="nofollow noreferrer">this paper</a> by Talathi and Vartak shows that the initial value of the recurrent weight matrix will strongly influence the performance of a recurrent neural network with reLU activation. Specifically, the authors demonstrate that a network of recurrent units with ReLU activation will perform best when the recurrent weight matrix is initialized so that it is positive definite, with the largest eigenvalue equal to one and all other eignenvalues less than one. Their explanation is that the way the network weights evolve over time is dependent on the initial condition of the network (shown in Figure 2 of Talathi and Vartak), and can lead to a few different cases:</p>

<p>Case 1: If all eigenvalues are one, then the network starts at a stable condition and does not evolve over time (Figure 2a)</p>

<p>Case 2: If all eigenvalues are less than one the network is attracted toward the origin and the network will always evolve toward a recurrent weight matrix of zero (Figure 2b).  </p>

<p>Case 3: If any eignevalues are greater than one the network does not have a stable attractor, and will "blow up" (Figure 2d).  </p>

<p>Case 4: If the recurrent weight matrix has one eigenvalue of one and the rest are less than one, then there can be a stable manifold that the network will evolve toward, and it can reach a stable, non-zero solution (Figure 2c). I don't know if this is guaranteed to be true for all problems, so I believe it'd be best to think of this as a necessary but not proven to be sufficient condition.</p>

<p>From the initial question, it sounds like @rksh's problem is the second case, and the network is attracted toward zero. Try initializing the weight matrix as Talathi and Vartak suggest and see if that addresses the problem.</p>
ReLU Function In A Recurrent Neural Network. Weights Become Infinity or Zero
<p>I am new to machine learning. I've read that the ReLU function is better than a sigmoid function for a recurrent neural network because of the vanishing gradient problem.</p>

<p>I'm trying to implement a very basic recurrent neural network with 3 input nodes, 10 hidden nodes and 3 output nodes.</p>

<p>There is the ReLU function at both input and hidden nodes and the softmax function for the output nodes.</p>

<p>However when I'm using the ReLU function after few epochs (less than 10) either the error gets to 0 or the error gets to infinity depending on whether the weight changes are added or subtracted from the original weights.</p>

<pre><code>weight = weight + gradient_decent #weights hits infinity
weight = weight - gradient decent #weights become 0
</code></pre>

<p>And also because it hits infinity it gives the following error,</p>

<pre><code>RuntimeWarning: invalid value encountered in maximum
  return np.maximum(x, 0)
</code></pre>

<p>However when I implement the sigmoid function the error nicely comes down. But because this is a simple example that is fine but if I use it on a bigger problem I am afraid I will hit with the vanishing gradient problem.</p>

<p>Is this caused by the small number of hidden nodes, how can I solve this issue? If you need the code sample please comment, not posting the code because it's too long.</p>

<p>Thank you.</p>