title: Keras LSTM: Bar plot of Feature Importance using SHAP
body: <p>I'm working on a LSTM for time series forecasting. Now I want to understand, which features impact the output the most and which ones aren't important. The goal is a bar plot like that using matplotlib in pycharm (reference: <a href="https://github.com/slundberg/shap" rel="nofollow noreferrer">https://github.com/slundberg/shap</a>):
  <a href="https://i.stack.imgur.com/hUZAW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hUZAW.png" alt="enter image description here"></a></p>
  
  <p>Here is my model:</p>
  
  <pre><code>model = Sequential()
  model.add(LSTM(num_neurons, activation='relu', input_shape=(train_x.shape[1], train_x.shape[2]), return_sequences=True))
  model.add(SeqWeightedAttention())
  model.add(Dropout(dropout))
  model.add(Dense(1))
  adam_opt = keras.optimizers.Adam(lr=adam_lr, beta_1=adam_b1, beta_2=adam_b2, amsgrad=False)
  model.compile(loss='mae', optimizer=adam_opt, metrics=[metrics.mae, metrics.mape, metrics.mse])
  history = model.fit(train_x, train_y, epochs=n_epoch, batch_size=batch_size, validation_data=(val_x, val_y), verbose=2, shuffle=False, callbacks=[early_stop, checkpoint])
  
  shap.initjs()
  background = train_x[np.random.choice(train_x.shape[0], 100, replace=False)]
  explainer = shap.DeepExplainer(model, background)
  shap_values = explainer.shap_values(val_x)
  shap.summary_plot(shap_values, train_x, plot_type="bar", matplotlib=True)
  plt.show()
  </code></pre>
  
  <p>And I get the following error calculating the shap_values (shap_values = ...):</p>
  
  <pre><code>ValueError: Dimension 2 in both shapes must be equal, but are 20 and 1. Shapes are [?,3,20] and [?,?,1]. for 'gradients/seq_weighted_attention_1/mul_grad/Select_1' (op: 'Select') with input shapes: [?,?,1], [?,3,20], [?,3,20].
  </code></pre>
  
  <p>Any ideas? Thanks in advance!</p>