<p>Faster convergence with a very high loss could possibly mean you are facing an exploding gradients problem. Try to use a much lower learning rate like 1e-5 or 1e-6. You can also try techniques like gradient clipping to limit your gradients in case of high learning rates.</p>

<p><strong>Answer 1</strong> </p>

<p>Another reason could be initialization of weights, try the below 3 methods:</p>

<ol>
<li>Method described in this paper <a href="https://arxiv.org/abs/1502.01852" rel="noreferrer">https://arxiv.org/abs/1502.01852</a></li>
<li>Xavier initialization</li>
<li>Random initialization</li>
</ol>

<p>For many cases 1st initialization method works the best.</p>

<p><strong>Answer 2</strong></p>

<p>You can try different optimizers like </p>

<ol>
<li>Momentum optimizer</li>
<li>SGD or Gradient descent</li>
<li>Adam optimizer</li>
</ol>

<p>The choice of your optimizer should be based on the choice of your loss function. For example: for a logistic regression problem with MSE as a loss function, gradient based optimizers will not converge.</p>

<p><strong>Answer 3</strong></p>

<p>How deep or wide your network should be is again fully dependent on which type of network you are using and what the problem is. </p>

<p>As you said you are using a sequential model using LSTM, to learn sequence on text. No doubt your choice of model is good for this problem you can also try 4-5 LSTMs.</p>

<p><strong>Answer 4</strong></p>

<p>If your gradients are going either 0 or infinite, it is called vanishing gradients or it simply means early convergence, try gradient clipping with proper learning rate and the first weight initialization technique.</p>

<p>I am sure this will definitely solve your problem.</p>
Python Keras LSTM learning converges too fast on high loss
<p>This is more of a deep learning conceptual problem, and if this is not the right platform I'll take it elsewhere.</p>

<p>I'm trying to use a Keras LSTM sequential model to learn sequences of text and map them to a numeric value (a regression problem).</p>

<p>The thing is, the learning always converges too fast on high loss (both training and testing). I've tried all possible hyperparameters, and I have a feeling it's a local minima issue that causes the model's high bias.</p>

<p>My questions are basically :</p>

<ol>
<li>How to initialize weights and bias given this problem?</li>
<li>Which optimizer to use?</li>
<li>How deep I should extend the network (I'm afraid that if I use a very deep network, the training time will be unbearable and the model variance will grow)</li>
<li>Should I add more training data?</li>
</ol>

<p>Input and output are normalized with minmax.</p>

<p>I am using SGD with momentum, currently 3 LSTM layers (126,256,128) and 2 dense layers (200 and 1 output neuron)</p>

<p>I have printed the weights after few epochs and noticed that <strong>many weights 
are zero and the rest are basically have the value of 1</strong> (or very close to it).</p>

<p>Here are some plots from tensorboard :<a href="https://i.stack.imgur.com/adC9M.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/adC9M.jpg" alt="enter image description here"></a></p>