<blockquote>
  <p>Could you please tell me which metrics and loss I need and if I need custom ones also?</p>
</blockquote>

<p>From the paper <a href="https://arxiv.org/pdf/1411.4389.pdf" rel="nofollow noreferrer">document </a> you can see the work is old: May 2016. </p>

<p>Please consider some recent work with more specifics. </p>

<p>This paper, provides no clue on LSTM specifics except the variable number of units they used, so you may try to find models with explained metrics.</p>
Right metrics and loss for averaging label probabilities across an RNN
<p>I am trying to reproduce that in Keras:</p>

<blockquote>
  <p>LRCN is trained to predict the
  video’s activity class at each time step. To produce a single
  label prediction for an entire video clip, we average the label
  probabilities—the outputs of the network’s softmax layer—
  across all frames and choose the most probable label.</p>
</blockquote>

<p>But I am quite new to LSTMs and am not sure about which metrics and loss function to use to replicate the method applied in the text above.
So far I have an LSTM RNN, which returns sequences and its outputs I feed into a time-distributed dense layer of 3 classes.</p>

<p>A "frame" corresponds to a timestep of the RNN and <code>return_sequences=True</code> will enable me to return prediction per frame.</p>

<p>Could you please tell me which metrics and loss I need and if I need custom ones also?</p>