title: How to interpret multi-class deep learning classifier by using SHAP?
body: <p>I trained a multi-class classifier in Keras on IRIS data set. I want to interpret my deep learning model by using <a href="https://github.com/slundberg/shap" rel="nofollow noreferrer">SHAP</a>. I use the following lines of code where model is my trained neural network classifier, <code>X1[train]</code> is my training input, and <code>X1[test]</code> is my test input:</p>
  
  <pre><code>import shap
  explainer = shap.DeepExplainer(model, X1[train])
  shap_values = explainer.shap_values(X1[test])
  </code></pre>
  
  <p>However, I get this error (I am fine when my <strong>softmax</strong> has one output in binary classification. The problem appears when softmax has <em>more than 1 output</em>):</p>
  
  <pre><code>ValueError: Unexpectedly found an instance of type `&lt;class 'numpy.ndarray'&gt;`. Expected a symbolic tensor instance.
  
  During handling of the above exception, another exception occurred:
  Layer sequential_96 was called with an input that isn't a symbolic tensor. Received type: &lt;class 'numpy.ndarray'&gt;. Full input:
  </code></pre>
  
  <p>How can I solve this problem and get SHAP values for a single class in multi class deep learning classifier?</p>