<p>I think your LSTM is not able to extract relevant features from the video frames in order to achieve a good accuracy.</p>

<p>The approach that usually gives the best results when dealing with images (or video frames) is extracting features with a stack of convolution + relu + max pooling layers (see <a href="https://arxiv.org/abs/1612.02903" rel="nofollow noreferrer">https://arxiv.org/abs/1612.02903</a> which is a survey on facial expression recognition, they all use convolutions to extract useful features from images).</p>

<p>These work best with 2-dimensional input, but i see that you represent a video frame with an array of size 2048 instead of a matrix. Usually images are represented with a shape similar to <code>(rows, cols, color_channels)</code>.</p>

<p>In your case, the input would have shape <code>(1, None, rows, cols, color_channels)</code>, then the convolutions would look something like this:</p>

<pre class="lang-py prettyprint-override"><code>from keras.layers import Input, LSTM, Conv2D, MaxPool2D, TimeDistributed, Flatten

x = Input(batch_shape=(1, None, rows, cols, color_channels), name='x')
convs = TimeDistributed(Conv2D(16, kernel_size=(3,3), activation='relu', padding='same'))(x)
convs = TimeDistributed(MaxPool2D(pool_size=(2,2)))(convs)
convs = TimeDistributed(Conv2D(32, kernel_size=(3,3), activation='relu', padding='same'))(convs)
convs = TimeDistributed(MaxPool2D(pool_size=(2,2)))(convs)
lstm_input = TimeDistributed(Flatten())(convs)
lstmR = LSTM(256, return_sequences=True, name='lstmR', stateful=True)(lstm_input)
lstmL = LSTM(256, return_sequences=True, go_backwards=True, name='lstmL', stateful=True)(lstm_input)
...
</code></pre>

<p>Where <code>TimeDistrubuted</code> applies the given layer to each time step.</p>
Use Stateful LSTM with mini-batching and input with variable time steps, in Keras?
<p>I'm new in Keras, and trying to implement this network
<a href="https://i.stack.imgur.com/MHyVi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MHyVi.png" alt="enter image description here"></a></p>

<p>this network takes a video frames as x = {x1,........,xT} where T is the number of the frames in the video and x is the visual features of the frames
of size 2048</p>

<p>i tried to use stateful LSTM as each sample have a number of frames as refereed <a href="http://philipperemy.github.io/keras-stateful-lstm/" rel="nofollow noreferrer">here</a></p>

<p>and this is my model</p>

<pre><code>x = Input(batch_shape=(1, None, 2048), name='x')
lstmR = LSTM(256, return_sequences=True, name='lstmR', stateful=True)(x)
lstmL = LSTM(256, return_sequences=True, go_backwards=True,name='lstmL', stateful=True)(x)
merge = merge([x, lstmR, lstmL], mode='concat', name='merge')
dense = Dense(256, activation='sigmoid', name='dense')(merge)
y = Dense(1, activation='sigmoid', name='y')(dense)
model = Model(input=x, output=y)
model.compile(loss='mean_squared_error',
          optimizer=SGD(lr=0.01),
          metrics=['accuracy'])
</code></pre>

<p>and tried to train the model using mini-batching</p>

<pre><code>for epoch in range(15):
    mean_tr_acc = []
    mean_tr_loss = []
    for i in range(nb_samples):
        x, y = get_train_sample(i)
        for j in range(len(x)):
            sample_x = x[j]
            tr_loss, tr_acc = model.train_on_batch(np.expand_dims(np.expand_dims(sample_x, axis=0), axis=0),np.expand_dims(y, axis=0))
            mean_tr_acc.append(tr_acc)
            mean_tr_loss.append(tr_loss)
        model.reset_states()
</code></pre>

<p>but it seems like the model cannot converge as it gives 0.3 accuracy</p>

<p>I also tried to do it with stateless LSTM with input shape (None,1024) but it didn't converge too</p>