<p>Here are some of the things that I would try next. 
(I am also an amateur. Please correct me if I am wrong)</p>

<ol>
<li>Try to extract <a href="https://machinelearningmastery.com/what-are-word-embeddings/" rel="nofollow noreferrer">vector representation</a> from the text. Try out word2vec, GloVe, FastText, ELMo. Extract vector representation and then feed them into the network. You could also create an <a href="https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/" rel="nofollow noreferrer">embedding layer</a> to help with that. This <a href="http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/" rel="nofollow noreferrer">blog</a> has more information.</li>
<li>256 Recurrent units might be too much. I think that one should never start with a huge network. Start small. See if you are underfitting. If yes, then go larger.</li>
<li>Switch out the optimizer. I find that Adam tends to overfit. I had better success with rmsprop and Adadelta.</li>
<li>Perhaps, <a href="https://duckduckgo.com/?q=attention%20is%20all%20you%20need&amp;t=osx&amp;ia=web" rel="nofollow noreferrer">attention is all you need?</a> Transformers have recently made massive contributions to NLP. Perhaps you could try <a href="https://github.com/philipperemy/keras-attention-mechanism" rel="nofollow noreferrer">implementing simple soft attention mechanism</a> in your network. Here is a <a href="https://www.youtube.com/watch?v=SysgYptB198" rel="nofollow noreferrer">nice video series</a> if you are not already familiar. An <a href="https://distill.pub/2016/augmented-rnns/#attentional-interfaces" rel="nofollow noreferrer">interactive research paper</a> on it.</li>
<li>CNN's are also <a href="https://arxiv.org/abs/1408.5882" rel="nofollow noreferrer">pretty dope</a> in NLP applications. Although they intuitively don't make any sense for text data (to most people). Perhaps you could try leveraging them, stack it, etc. Play around. Here is a <a href="https://arxiv.org/pdf/1510.03820.pdf" rel="nofollow noreferrer">guide</a> on how to use it for sentence classification. I know, your domain is different. But I think the intuition carries over. :)</li>
</ol>
Problem with text prediction LSTM neural networks
<p>I'm trying to do text prediction using recurrent neural networks (LSTM) with dataset from books. It doesn't matter how much I try changing layers size or other parameters, it always overfits. </p>

<p>I've been trying changing amount of layers, amount of units in LSTM layer, regularization, normalization, batch_size, shuffle training data/validation data, change dataset to bigger. For now I try with ~140kb txt book. I have also tried 200kb, 1mb, 5mb.</p>

<p>Creating training/validation data:</p>

<pre><code>sequence_length = 30

x_data = []
y_data = []

for i in range(0, len(text) - sequence_length, 1):
    x_sequence = text[i:i + sequence_length]
    y_label = text[i + sequence_length]

    x_data.append([char2idx[char] for char in x_sequence])
    y_data.append(char2idx[y_label])

X = np.reshape(x_data, (data_length, sequence_length, 1))
X = X/float(vocab_length)
y = np_utils.to_categorical(y_data)

# Split into training and testing set, shuffle data
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2, shuffle=False)

# Shuffle testing set
X_test, y_test = shuffle(X_test, y_test, random_state=0)
</code></pre>

<p>Creating model:</p>

<pre><code>model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True, recurrent_initializer='glorot_uniform', recurrent_dropout=0.3))
model.add(LSTM(256, return_sequences=True, recurrent_initializer='glorot_uniform', recurrent_dropout=0.3))
model.add(LSTM(256, recurrent_initializer='glorot_uniform', recurrent_dropout=0.3))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
</code></pre>

<p><a href="https://i.stack.imgur.com/RC3ME.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RC3ME.png" alt="enter image description here"></a>
Compile model:</p>

<pre><code>model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
</code></pre>

<p>I get following characteristics:
<a href="https://i.stack.imgur.com/rpk0V.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rpk0V.png" alt="enter image description here"></a></p>

<p>I don't know what to do about this overfitting, because I am searching internet, trying many things but none of them seems to work.</p>

<p>How could I get better results? These prediction doesn't seem to be good right now.</p>