<p>For CNN-RNN, some promising things to try:</p>

<ul>
<li><strong>Conv1D layers</strong>: <code>activation='relu'</code>, <code>kernel_initializer='he_normal'</code></li>
<li><strong>LSTM layer</strong>: <code>activation='tanh'</code>, and <code>recurrent_dropout=.1, .2, .3</code></li>
<li><strong>Optimizer</strong>: <code>Nadam</code>, <code>lr=2e-4</code> (Nadam may significantly outperform all other optimizers for RNNs)</li>
<li><strong>batch_size</strong>: lower it. Unless you have 200+ batches in total, set <code>batch_size=32</code>; lower batch size better exploits the Stochastic mechanism of the optimizer and can improve generalization</li>
<li><strong>Dropout</strong>: right after second <code>Conv1D</code>, with a rate <code>.1, .2</code> - or, after first <code>Conv1D</code>, with a rate <code>.25, .3</code>, but <em>only</em> if you use SqueezeExcite (see below), else <code>MaxPooling</code> won't work as well</li>
<li><a href="https://arxiv.org/abs/1709.01507" rel="nofollow noreferrer">SqueezeExcite</a>: shown to enhance all CNN performance across a large variety of tasks; Keras implementation you can use below</li>
<li><strong>BatchNormalization</strong>: while your model isn't large, it's still deep, and may benefit from one BN layer right after second <code>Conv1D</code></li>
<li><strong>L2 weight decay</strong>: on <em>first</em> <code>Conv1D</code>, to prevent it from memorizing the input; try <code>1e-5, 1e-4</code>, e.g. <code>kernel_regularizer=l2(1e-4) # from keras.regularizers import l2</code></li>
<li><strong>Preprocessing</strong>: make sure all data is normalized (or standardized if time-series), and batches are shuffled each epoch</li>
</ul>

<pre><code>def SqueezeExcite(_input):
    filters = _input._keras_shape[-1]

    se = GlobalAveragePooling1D()(_input)
    se = Reshape((1, filters))(se)
    se = Dense(filters//16,activation='relu',   
               kernel_initializer='he_normal', use_bias=False)(se)
    se = Dense(filters,    activation='sigmoid',
               kernel_initializer='he_normal', use_bias=False)(se)

    return multiply([_input, se])
</code></pre>

<pre><code># Example usage
x = Conv1D(filters=64, kernel_size=4, activation='relu', kernel_initializer='he_normal')(x)
x = SqueezeExcite(x) # place after EACH Conv1D
</code></pre>
Training and Testing accuracy not increasing for a CNN followed by a RNN for signature verification
<p>I'm currently working on online signature verification. The dataset has a variable shape of (x, 7) where x is the number of points a person used to sign their signature. I have the following model:</p>

<pre><code>    model = Sequential()
    #CNN
    model.add(Conv1D(filters=64, kernel_size=3, activation='sigmoid', input_shape=(None, 7)))
    model.add(MaxPooling1D(pool_size=3))
    model.add(Conv1D(filters=64, kernel_size=2, activation='sigmoid'))

    #RNN
    model.add(Masking(mask_value=0.0))
    model.add(LSTM(8))
    model.add(Dense(2, activation='softmax'))

    opt = Adam(lr=0.0001)
    model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
    model.summary()

    print(model.fit(x_train, y_train, epochs=100, verbose=2, batch_size=50))

    score, accuracy = model.evaluate(x_test,y_test, verbose=2)
    print(score, accuracy)
</code></pre>

<p>I know it may not be the best model but this is the first time I'm building a neural network. I have to use a CNN and RNN as it is required for my honours project. At the moment, I achieve 0.5142 as the highest training accuracy and 0.54 testing accuracy. I have tried increasing the number of epochs, changing the activation function, add more layers, moving the layers around, changing the learning rate and changing the optimizer. </p>

<p>Please share some advice on changing my model or dataset. Any help is much appreciated.  </p>