<p>Adding few more ideas in answer pointed by <strong>bhaskar</strong>, which are used to handle this problem.</p>

<p>You can used <strong>Attention</strong> mechanism, which is used to deal with long term dependencies. Because for a long sequence, it certainly forget information or its next prediction may not depend on all the sequence information, it has in its cell. So <code>attention mechanism helps to find the reasonable weights for the characters, it depend on.</code> For more info you can check this <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwj5yc2I7uzfAhUCYo8KHRWzBkIQFjAAegQIBxAB&amp;url=https%3A%2F%2Farxiv.org%2Fabs%2F1706.03762&amp;usg=AOvVaw2ceXGQohV5Kx51VSkfkG08" rel="nofollow noreferrer">link</a></p>

<p>There is potentially lots of research on this problem. <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=7&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjW-JWv7uzfAhUGaI8KHWIdCNwQFjAGegQIAhAC&amp;url=https%3A%2F%2Farxiv.org%2Fpdf%2F1803.00144&amp;usg=AOvVaw2S-aKSTJS-tomj5IGFiVgT" rel="nofollow noreferrer">This</a> is very recent paper on this problem.</p>

<p>You can also break the sequence and use <code>seq2seq</code> model, which encode the features into low dims space and then decoder will extract it . This is <a href="https://github.com/zhangruiskyline/DeepLearning/blob/master/doc/RNN.md" rel="nofollow noreferrer">short-article</a> on this.</p>

<p>My personal advice is to break the sequence and then train it, because sliding window on the complete sequence is pretty much able to find the correlation between each sequence.</p>
Practical advice on dealing with very long inputs using LSTM model?
<p>I built a character-level LSTM model on text data, but ultimately I'm looking to apply this model on very long text documents (such as a novel) where it's important to understand contextual information, such as where in the novel it's in.  </p>

<p>For these large-scale NLP tasks, is the data usually cut into smaller pieces and concatenated with metadata - such as position within the document, detected topic, etc. - to be fed into the model?  Or are there more elegant techniques?</p>