<p>The <strong>Attention Mechanism</strong> is used for this exactly; programmatic implementation isn't simple, but use-ready repositories exist - see below. Example output below.</p>

<p>As to what attention 'is', see <a href="https://stats.stackexchange.com/questions/344508/what-are-attention-mechanisms-exactly">this SE answer</a>, and/or <a href="https://www.quora.com/What-is-exactly-the-attention-mechanism-introduced-to-RNN-recurrent-neural-network-It-would-be-nice-if-you-could-make-it-easy-to-understand" rel="nofollow noreferrer">this Quora answer</a>; in a nutshell, it's a means of identifying the most 'important' timesteps, effectively mapping out a temporal 'heatmap'.</p>

<ul>
<li><a href="https://github.com/albermax/innvestigate" rel="nofollow noreferrer">iNNvestigate</a>, classifier introspection (first image below; can be applied to timeseries)</li>
<li><a href="https://raghakot.github.io/keras-vis/visualizations/activation_maximization/" rel="nofollow noreferrer">Saliency maps</a>, extracted features introspection</li>
<li><a href="https://stackoverflow.com/questions/58356868/how-visualize-attention-lstm-using-keras-self-attention-package/58357581#58357581">LSTM/CNN Visualization</a>, simple function (second image below)</li>
<li><a href="https://github.com/ningshixian/LSTM_Attention" rel="nofollow noreferrer">LSTM_Attention</a> - includes research paper-specific implementations.</li>
</ul>

<p>Lastly, as a tip, ditch LSTMs for <a href="https://arxiv.org/abs/1803.04831" rel="nofollow noreferrer">IndRNNs</a>; where former struggles w/ 800-1000 timesteps, latter's shown to succeed w/ 5000+. Features are also more interpretable, as each channel is independent, absent LSTM-type gating mechanisms. Though if speed is important, there is no <a href="https://keras.io/layers/recurrent/#cudnnlstm" rel="nofollow noreferrer"><code>CuDNNIndRNN</code></a>.</p>

<p><hr>
<img src="https://raw.githubusercontent.com/albermax/innvestigate/master/examples/images/analysis_grid.png" width="700">
<img src="https://i.stack.imgur.com/l26NF.png" width="700"></p>
How to identify important features in LSTM
<p>I have a binary classification problem where for each data point I have 3 time-series as follows.</p>

<pre><code>data_point,   time_series1,      time_series2,      time_series3,  label
d1,         [0.1, ....., 0.5], [0.8, ....., 0.6], [0.8, ....., 0.8], 1
and so on
</code></pre>

<p>I am using the following code to perform my binary classification.</p>

<pre><code>model = Sequential()
model.add(LSTM(100, input_shape=(25,4)))
model.add(Dense(50))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
</code></pre>

<p>Since, currently I am considering my classification as a black-box task, I would like to dig deeper and see what happens inside.</p>

<p>More specifically, I would like to know the imporatant features used by LSTM to classify my datapoints. More importantly I want to answer the following questions;</p>

<ul>
<li>Which time series (i.e. <code>time_series1</code>, <code>time_series2</code>,
<code>time_series3</code>) was most influenced in the classification </li>
<li>What are features extracted from the most influenced timeseries?</li>
</ul>

<p>I am happy to provide more details if needed.</p>