<p>There are many visualization methods. Each of these methods has its strengths and weaknesses.</p>

<p>However, you have to keep in mind that the methods partly visualize different things. Here is a short overview based on this <a href="https://arxiv.org/abs/1705.05598" rel="nofollow noreferrer">paper</a>.
You can distinguish between three main visualization groups: </p>

<ul>
<li><strong>Functions</strong> (gradients, saliency map): These methods visualize how a change in input space affects the prediction</li>
<li><strong>Signal</strong> (deconvolution, Guided BackProp, PatternNet): the signal (reason for a neuron's activation) is visualized. So this visualizes what pattern caused the activation of a particular neuron.</li>
<li><strong>Attribution</strong> (LRP, Deep Taylor Decomposition, PatternAttribution): these methods visualize how much a single pixel contributed to the prediction. As a result you get a heatmap highlighting which pixels of the input image most strongly contributed to the classification.</li>
</ul>

<p>Since you are asking how much a pixel has contributed to the classification, you should use methods of attribution. Nevertheless, the other methods also have their right to exist.</p>

<p>One nice toolbox for visualizing heatmaps is <a href="https://github.com/albermax/innvestigate" rel="nofollow noreferrer">iNNvestigate</a>.
This toolbox contains the following methods:</p>

<ul>
<li><a href="https://arxiv.org/abs/1706.03825" rel="nofollow noreferrer">SmoothGrad</a></li>
<li><a href="https://arxiv.org/abs/1311.2901" rel="nofollow noreferrer">DeConvNet</a></li>
<li><a href="https://arxiv.org/abs/1412.6806" rel="nofollow noreferrer">Guided BackProp</a></li>
<li><a href="https://arxiv.org/abs/1705.05598" rel="nofollow noreferrer">PatternNet</a></li>
<li><a href="https://arxiv.org/abs/1705.05598" rel="nofollow noreferrer">PatternAttribution</a></li>
<li>Occlusion</li>
<li><a href="https://arxiv.org/abs/1605.01713" rel="nofollow noreferrer">Input times Gradient</a></li>
<li><a href="https://arxiv.org/abs/1703.01365" rel="nofollow noreferrer">Integrated Gradients</a></li>
<li><a href="https://www.sciencedirect.com/science/article/pii/S0031320316303582?via%3Dihub" rel="nofollow noreferrer">Deep Taylor</a></li>
<li><a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140" rel="nofollow noreferrer">LRP</a></li>
<li><a href="https://openreview.net/forum?id=Sy21R9JAW" rel="nofollow noreferrer">DeepLift</a></li>
</ul>
CNN attention/activation maps
<p>What are common techniques for finding which parts of images contribute most to image classification <strong>via convolutional neural nets</strong>?</p>

<p>In general, suppose we have 2d matrices with float values between 0 and 1 as entires. Each matrix is associated with a label (single-label, multi-class) and the goal is to perform classification via (Keras) 2D CNN's.</p>

<p>I'm trying to find methods to extract relevant subsequences of rows/columns that contribute most to classification. </p>

<p>Two examples:</p>

<p><a href="https://github.com/jacobgil/keras-cam" rel="noreferrer">https://github.com/jacobgil/keras-cam</a></p>

<p><a href="https://github.com/tdeboissiere/VGG16CAM-keras" rel="noreferrer">https://github.com/tdeboissiere/VGG16CAM-keras</a></p>

<p>Other examples/resources with an eye toward Keras would be much appreciated.</p>

<p>Note my datasets are not actual images, so using methods with ImageDataGenerator might not directly apply in this case.</p>