<p>Good question. The lack of interpretability of NN models is one pain the ML/NN community has been struggling with.</p>

<p>One recent approach that has been receiving attention is the <a href="https://arxiv.org/abs/1602.04938" rel="noreferrer">LIME paper</a> (Ribeiro et al, KDD'16). 
Here's a relevant excerpt from the abstract:</p>

<ul>
<li><em>"In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction"</em>.</li>
</ul>

<p>There's also a GitHub <a href="https://github.com/marcotcr/lime" rel="noreferrer">repository</a> (Python, yay!).</p>

<p>(If you do try LIME, please share your experience in the question comments..)</p>
Most important features in MLPClassifier in Sklearn
<p>I would like to know if there is any way to visualize or find the most important/contributing features after fitting a MLP classifier in Sklearn.</p>

<p>Simple example:</p>

<pre><code>import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import LeaveOneOut
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline


data= pd.read_csv('All.csv', header=None)
X, y = data.iloc[0:, 0:249].values, data.iloc[0:,249].values

sc = StandardScaler()
mlc = MLPClassifier(activation = 'relu', random_state=1,nesterovs_momentum=True)
loo = LeaveOneOut()
pipe = make_pipeline(sc, mlc)

parameters = {"mlpclassifier__hidden_layer_sizes":[(168,),(126,),(498,),(166,)],"mlpclassifier__solver" : ('sgd','adam'), "mlpclassifier__alpha": [0.001,0.0001],"mlpclassifier__learning_rate_init":[0.005,0.001] }
clf = GridSearchCV(pipe, parameters,n_jobs= -1,cv = loo)
clf.fit(X, y)

model = clf.best_estimator_
print("the best model and parameters are the following: {} ".format(model))
</code></pre>