<p><a href="https://arxiv.org/abs/1502.03167" rel="nofollow">Batch normalisation</a> is now well accepted as a general learning facilitator and régularizer.</p>

<p>Here is a Tensorflow implementation of a batch normalized LSTM Cell: <a href="https://github.com/OlavHN/bnlstm/blob/master/lstm.py" rel="nofollow">https://github.com/OlavHN/bnlstm/blob/master/lstm.py</a></p>

<p>This implementation Explained in the article here : <a href="http://olavnymoen.com/2016/07/07/rnn-batch-normalization" rel="nofollow">Batch normalized LSTM for Tensorflow</a> </p>

<p>It is applying the principles from the paper: <a href="https://arxiv.org/abs/1603.09025" rel="nofollow">Recurrent Batch Normalization (arXiv)</a></p>
How do I tune state size for a Tensorflow RNN / LSTM, or more generally reduce overfitting?
<p>I have a classifier with ~1100 features and 60k samples of training data. I create an RNN with 1100 LSTMcells, and it classifies all my training data correctly, and then underperforms on the test data.</p>

<p>If I had a very large feed-forward NN I think it would behave similarly, and one would reduce size of hidden layer(s), add regularization, dropout, etc. to reduce overfitting.</p>

<p>How would I do the same for the RNN/LSTM? (added dropout but don't see a way to add regularization or especially control the LSTM state size - seems to default to input size which is probably too large)</p>

<p>I see that there was an input_size parameter is now deprecated and unused.</p>

<p><a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard5/tf.nn.rnn_cell.LSTMCell.md" rel="nofollow">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard5/tf.nn.rnn_cell.LSTMCell.md</a></p>

<p>I see references in that doc to </p>

<pre><code>{#LSTMCell.init}
{#LSTMCell.output_size}
{#LSTMCell.state_size}
</code></pre>

<p>but how does one use them? the simple tutorial examples just use the defaults, which result in overfitting.</p>

<p>If there is some other way to discover and tune hyperparameters I'm not seeing it.</p>