<p>There is nothing wrong with your training or sampling - this is the expected behavior for a "pure" LSTM network. To model the variance in your data, don't make the network predict the values at the next time step directly. Rather, your network should give you a probability distribution over the possible values for the next time step, from which you can then sample. </p>

<p>Two examples of how you can do this:</p>

<ul>
<li>Discrete data, e.g. text: stack a softmax layer on top of the LSTM, which gives you the probabilites for each letter, then sample from these probabilites - this is also implemented in Karparthy's infamous <a href="https://github.com/karpathy/char-rnn" rel="nofollow">char-rnn</a>, see the paragraph "Temperature"</li>
<li>Continuous data, e.g. timeseries: make the network predict the parameters of a mixture distribution (i.e. a linear combination of Gaussians), then sample from this - I very much recommend the section on handwriting prediction in <a href="http://arxiv.org/abs/1308.0850" rel="nofollow">Graves 2013</a>, or you can have a look at chapter 5 of <a href="https://github.com/jrieke/lstm-biology/blob/master/Project%20Report.pdf" rel="nofollow">this report</a> I recently wrote for a research project</li>
</ul>
Why does my LSTM stablize to fix point?
<p>I have trained a LSTM by a sequence, and try to test if it can synthesize some output sequence, but interestingly and unfortunately, it very quickly, i.e, after 2 time steps, stablizes to a fix output, meaning a sequence of exact same values.</p>

<p>Now I have changed the initialization, but the outputs are always the same after 2 steps. What might be wrong in the training or sampling?</p>

<p>Sorry that I cannot give more context, because the whole program is a big too large to post here.</p>