<blockquote>
<p>Is there any way to decompose a Keras CNN model and/or it's weights to glean how it's arriving at it's decision?</p>
</blockquote>
<p>You can look at the <a href="https://arxiv.org/abs/1610.02391" rel="nofollow noreferrer">Grad-CAM algorithm</a> to see where the neural network was looking at before taking its final decision. Here's an <a href="https://keras.io/examples/vision/grad_cam/" rel="nofollow noreferrer">implementation in Keras</a> using a pre-trained model.</p>
<pre><code>import numpy as np
import tensorflow as tf
from tensorflow import keras
from IPython.display import Image, display
import matplotlib.pyplot as plt
import matplotlib.cm as cm

model_builder = keras.applications.xception.Xception
img_size = (299, 299)
preprocess_input = keras.applications.xception.preprocess_input
decode_predictions = keras.applications.xception.decode_predictions

last_conv_layer_name = &quot;block14_sepconv2_act&quot;
classifier_layer_names = [
    &quot;avg_pool&quot;,
    &quot;predictions&quot;,
]

img_path = keras.utils.get_file(
    &quot;african_elephant.jpg&quot;, &quot;https://i.imgur.com/Bvro0YD.png&quot;
)

display(Image(img_path))


def get_img_array(img_path, size):
    img = keras.preprocessing.image.load_img(img_path, target_size=size)
    array = keras.preprocessing.image.img_to_array(img)
    array = np.expand_dims(array, axis=0)
    return array


def make_gradcam_heatmap(
    img_array, model, last_conv_layer_name, classifier_layer_names
):

    last_conv_layer = model.get_layer(last_conv_layer_name)
    last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)

    classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])
    x = classifier_input
    for layer_name in classifier_layer_names:
        x = model.get_layer(layer_name)(x)
    classifier_model = keras.Model(classifier_input, x)

    with tf.GradientTape() as tape:
        last_conv_layer_output = last_conv_layer_model(img_array)
        tape.watch(last_conv_layer_output)
        preds = classifier_model(last_conv_layer_output)
        top_pred_index = tf.argmax(preds[0])
        top_class_channel = preds[:, top_pred_index]

    grads = tape.gradient(top_class_channel, last_conv_layer_output)

    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))

    last_conv_layer_output = last_conv_layer_output.numpy()[0]
    pooled_grads = pooled_grads.numpy()
    for i in range(pooled_grads.shape[-1]):
        last_conv_layer_output[:, :, i] *= pooled_grads[i]

    heatmap = np.mean(last_conv_layer_output, axis=-1)
    heatmap = np.maximum(heatmap, 0) / np.max(heatmap)
    return heatmap

img_array = preprocess_input(get_img_array(img_path, size=img_size))

model = model_builder(weights=&quot;imagenet&quot;)

preds = model.predict(img_array)
print(&quot;Predicted:&quot;, decode_predictions(preds, top=1)[0])

heatmap = make_gradcam_heatmap(
    img_array, model, last_conv_layer_name, classifier_layer_names
)

img = keras.preprocessing.image.load_img(img_path)
img = keras.preprocessing.image.img_to_array(img)

heatmap = np.uint8(255 * heatmap)

jet = cm.get_cmap(&quot;jet&quot;)

jet_colors = jet(np.arange(256))[:, :3]
jet_heatmap = jet_colors[heatmap]

jet_heatmap = keras.preprocessing.image.array_to_img(jet_heatmap)
jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))
jet_heatmap = keras.preprocessing.image.img_to_array(jet_heatmap)

superimposed_img = jet_heatmap * 0.4 + img
superimposed_img = keras.preprocessing.image.array_to_img(superimposed_img)

save_path = &quot;elephant_cam.jpg&quot;
superimposed_img.save(save_path)

display(Image(save_path))
</code></pre>
<p><a href="https://i.stack.imgur.com/olpPF.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/olpPF.jpg" alt="enter image description here" /></a>
<a href="https://i.stack.imgur.com/nbKZl.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nbKZl.jpg" alt="enter image description here" /></a></p>
Is there a way to decompose a CNN model to provide prediction insights to humans?
<p>I have a CNN that after a lot of work is now performing multiclass (8) classification at 99% accuracy.</p>
<p>While the classification itself has a lot of value, going to a prediction engine would be a game changer.</p>
<p>The catch is that the ability to predict is needed by a human in real life (IRL), not in being processed by a computer.</p>
<p>In this case the CNN is able to classify things faster than the human. It would be significant if the model could provide insights into how it is classifying things</p>
<p>Is there any way to decompose a Keras CNN model and/or it's weights to glean how it's arriving at it's decision?  I don't believe so, but would hate not to ask and find out it's possible.</p>
<p>It's not that I'm looking for it to be exact, but if I can find one or two things that heavily influence the prediction/classification that a human being could key on, that could be significant.</p>
<p>Thoughts?</p>