<p><em>"Several methods for finding saliency have been described by other authors. Among them are sensitivity
based approaches [4, 5, 6], deconvolution based ones [7, 8], or more complex ones like
layer-wise relevance propagation (LRP) [9]."</em></p>

<p>source : <a href="https://arxiv.org/pdf/1704.07911.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1704.07911.pdf</a></p>

<p>They are doing what you want, but with a CNN, maybe you should go from a MLP to a CNN, that would seem appropriate for medical imaging classification.</p>

<p>Or maybe this paper would fit better:</p>

<p><a href="https://www.researchgate.net/publication/222561214_Illuminating_the_black_box_A_randomization_approach_for_understanding_variable_contributions_in_artificial_neural_networks" rel="nofollow noreferrer">Randomization approach for understanding variable contributions in artificial neural networks</a></p>
Interpreting internal features of Multiplayer Perceptrons
<p>I've trained a multiplayer perceptron on a medical imaging classification task (classifying whether an ultrasound scanning image belongs to the healthy or disease condition). The network consists of 2 fully connected hidden layers and 1 output unit. I then want to examine the weights to see which features in the images (e.g., clusters of pixels) are the most important for the network to distinguish between different classes. Since my network has two layers of hidden weights, how do I use these weights to quantify the importance of each image pixel? Could someone experienced with this point me to the right literature? Thanks.</p>