<p><em>Model interpretability</em> is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in  various tasks, plus the necessity of algorithmic fairness &amp; accountability...</p>
<p>Apart from the intense theoretical research, there have been some toolboxes &amp; libraries on a <strong>practical</strong> level lately, both for neural networks as well as for other general ML models; here is a partial list which arguably should keep you busy for some time:</p>
<ul>
<li><p>The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (<a href="http://www.jmlr.org/papers/v17/15-618.html" rel="nofollow noreferrer">paper</a>, <a href="http://heatmapping.org/" rel="nofollow noreferrer">project page</a>, <a href="https://github.com/sebastian-lapuschkin/lrp_toolbox" rel="nofollow noreferrer">code</a>, <a href="https://github.com/VigneshSrinivasan10/interprettensor" rel="nofollow noreferrer">TF Slim wrapper</a>)</p>
</li>
<li><p>FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (<a href="http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-box-predictive-models.html" rel="nofollow noreferrer">blog post</a>, <a href="https://arxiv.org/abs/1611.04967" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/adebayoj/fairml" rel="nofollow noreferrer">code</a>)</p>
</li>
<li><p>LIME: Local Interpretable Model-agnostic Explanations (<a href="https://arxiv.org/abs/1602.04938" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/marcotcr/lime" rel="nofollow noreferrer">code</a>, <a href="https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime" rel="nofollow noreferrer">blog post</a>, <a href="https://cran.r-project.org/web/packages/lime/index.html" rel="nofollow noreferrer">R port</a>)</p>
</li>
<li><p><a href="https://arxiv.org/abs/1602.07043" rel="nofollow noreferrer">Black Box Auditing</a> and <a href="https://arxiv.org/abs/1412.3756" rel="nofollow noreferrer">Certifying and Removing Disparate Impact</a> (authors' <a href="https://github.com/algofairness/BlackBoxAuditing" rel="nofollow noreferrer">Python code</a>)</p>
</li>
<li><p>A recent (November 2017) paper by Geoff Hinton, <a href="https://arxiv.org/abs/1711.09784" rel="nofollow noreferrer">Distilling a Neural Network Into a Soft Decision Tree</a>, with various independent <a href="https://paperswithcode.com/paper/distilling-a-neural-network-into-a-soft" rel="nofollow noreferrer">PyTorch implementations</a></p>
</li>
<li><p>SHAP: A Unified Approach to Interpreting Model Predictions (<a href="https://arxiv.org/abs/1705.07874" rel="nofollow noreferrer">paper</a>, authors' <a href="https://github.com/slundberg/shap" rel="nofollow noreferrer">Python code</a>, <a href="https://github.com/redichh/ShapleyR" rel="nofollow noreferrer">R package</a>)</p>
</li>
<li><p>Interpretable Convolutional Neural Networks (<a href="https://arxiv.org/abs/1710.00935" rel="nofollow noreferrer">paper</a>, authors' Matlab <a href="https://github.com/zqs1022/interpretableCNN" rel="nofollow noreferrer">code</a>)</p>
</li>
<li><p>Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (<a href="https://github.com/tensorflow/lucid" rel="nofollow noreferrer">code</a>; papers: <a href="https://distill.pub/2017/feature-visualization/" rel="nofollow noreferrer">Feature Visualization</a>, <a href="https://distill.pub/2018/building-blocks/" rel="nofollow noreferrer">The Building Blocks of Interpretability</a>)</p>
</li>
<li><p>Transparecy-by-Design (TbD) networks (<a href="https://arxiv.org/abs/1803.05268" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/davidmascharka/tbd-nets" rel="nofollow noreferrer">code</a>, <a href="https://mybinder.org/v2/gh/davidmascharka/tbd-nets/binder?filepath=full-vqa-example.ipynb" rel="nofollow noreferrer">demo</a>)</p>
</li>
<li><p>SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (<a href="https://arxiv.org/abs/1706.05806" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/google/svcca" rel="nofollow noreferrer">code</a>, <a href="https://ai.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html" rel="nofollow noreferrer">Google blog post</a>)</p>
</li>
<li><p>TCAV: Testing with Concept Activation Vectors (<a href="https://arxiv.org/abs/1711.11279" rel="nofollow noreferrer">ICML 2018 paper</a>, <a href="https://github.com/tensorflow/tcav" rel="nofollow noreferrer">Tensorflow code</a>)</p>
</li>
<li><p>Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (<a href="https://arxiv.org/abs/1610.02391" rel="nofollow noreferrer">paper</a>, authors' <a href="https://github.com/ramprs/grad-cam" rel="nofollow noreferrer">Torch code</a>, <a href="https://github.com/Ankush96/grad-cam.tensorflow" rel="nofollow noreferrer">Tensorflow code</a>, <a href="https://github.com/meliketoy/gradcam.pytorch" rel="nofollow noreferrer">PyTorch code</a>, Keras <a href="http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb" rel="nofollow noreferrer">example notebook</a>)</p>
</li>
<li><p>Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (<a href="http://netdissect.csail.mit.edu/" rel="nofollow noreferrer">project page</a>, <a href="https://github.com/CSAILVision/NetDissect" rel="nofollow noreferrer">Caffe code</a>, <a href="https://github.com/CSAILVision/NetDissect-Lite" rel="nofollow noreferrer">PyTorch port</a>)</p>
</li>
<li><p>GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (<a href="https://gandissect.csail.mit.edu/" rel="nofollow noreferrer">project page</a>, with links to paper &amp; code)</p>
</li>
<li><p>Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (<a href="https://arxiv.org/abs/1811.08011" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/gudovskiy/e2x" rel="nofollow noreferrer">code</a>)</p>
</li>
<li><p>Anchors: High-Precision Model-Agnostic Explanations (<a href="https://homes.cs.washington.edu/%7Emarcotcr/aaai18.pdf" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/marcotcr/anchor" rel="nofollow noreferrer">code</a>)</p>
</li>
<li><p>Diverse Counterfactual Explanations (DiCE) by Microsoft (<a href="https://arxiv.org/abs/1905.07697" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/microsoft/dice" rel="nofollow noreferrer">code</a>, <a href="https://www.microsoft.com/en-us/research/blog/open-source-library-provides-explanation-for-machine-learning-through-diverse-counterfactuals/" rel="nofollow noreferrer">blog post</a>)</p>
</li>
<li><p>Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (<a href="https://arxiv.org/abs/2008.02312" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/Fu0511/XGrad-CAM" rel="nofollow noreferrer">code</a>)</p>
</li>
</ul>
<p>As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is an (again, partial) list for Python stuff:</p>
<ul>
<li>The ELI5 Python library (<a href="https://github.com/TeamHG-Memex/eli5" rel="nofollow noreferrer">code</a>, <a href="https://eli5.readthedocs.io/en/latest/" rel="nofollow noreferrer">documentation</a>)</li>
<li>The What-If tool by Google, a brand new (September 2018) feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (<a href="https://pair-code.github.io/what-if-tool/" rel="nofollow noreferrer">project page</a>, <a href="https://github.com/pair-code/what-if-tool" rel="nofollow noreferrer">code</a>, <a href="https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html" rel="nofollow noreferrer">blog post</a>)</li>
<li>tf-explain - interpretability methods as Tensorflow 2.0 callbacks (<a href="https://github.com/sicara/tf-explain" rel="nofollow noreferrer">code</a>, <a href="https://tf-explain.readthedocs.io/en/latest/" rel="nofollow noreferrer">docs</a>, <a href="https://blog.sicara.com/tf-explain-interpretability-tensorflow-2-9438b5846e35" rel="nofollow noreferrer">blog post</a>)</li>
<li>InterpretML by Microsoft (<a href="https://interpret.ml/" rel="nofollow noreferrer">homepage</a>, <a href="https://github.com/Microsoft/interpret" rel="nofollow noreferrer">code</a> still in alpha, <a href="https://arxiv.org/abs/1909.09223" rel="nofollow noreferrer">paper</a>)</li>
<li>Captum by Facebook AI - model interpetability for Pytorch (<a href="https://captum.ai/" rel="nofollow noreferrer">homepage</a>, <a href="https://github.com/pytorch/captum" rel="nofollow noreferrer">code</a>, <a href="https://ai.facebook.com/blog/open-sourcing-captum-a-model-interpretability-library-for-pytorch/" rel="nofollow noreferrer">intro blog post</a>)</li>
<li>Skater, by Oracle (<a href="https://github.com/oracle/Skater" rel="nofollow noreferrer">code</a>, <a href="https://oracle.github.io/Skater/" rel="nofollow noreferrer">docs</a>)</li>
<li>Alibi, by SeldonIO (<a href="https://github.com/SeldonIO/alibi" rel="nofollow noreferrer">code</a>, <a href="https://docs.seldon.io/projects/alibi/en/stable/" rel="nofollow noreferrer">docs</a>)</li>
<li><a href="http://aix360.mybluemix.net/" rel="nofollow noreferrer">AI Explainability 360</a>, by IBM (<a href="https://github.com/IBM/AIX360" rel="nofollow noreferrer">code</a>, <a href="https://www.ibm.com/blogs/research/2019/08/ai-explainability-360/" rel="nofollow noreferrer">blog post</a>)</li>
</ul>
<p>See also:</p>
<ul>
<li><p><a href="https://christophm.github.io/interpretable-ml-book/" rel="nofollow noreferrer">Interpretable Machine Learning</a>, an online Gitbook by Christoph Molnar with <a href="https://github.com/christophM/iml" rel="nofollow noreferrer">R code</a> available</p>
</li>
<li><p><a href="https://pbiecek.github.io/ema/" rel="nofollow noreferrer">Explanatory Model Analysis</a>, another online book by Przemyslaw Biecek and Tomasz Burzykowski, with both R &amp; Python code snippets</p>
</li>
<li><p>A <a href="https://twitter.com/ledell/status/995930308947140608" rel="nofollow noreferrer">Twitter thread</a>, linking to several interpretation tools available for R.</p>
</li>
<li><p>A short (4 hrs) online course by Dan Becker at Kaggle, <a href="https://www.kaggle.com/learn/machine-learning-explainability" rel="nofollow noreferrer">Machine Learning Explainability</a>, and the accompanying <a href="https://towardsdatascience.com/why-model-explainability-is-the-next-data-science-superpower-b11b6102a5e0" rel="nofollow noreferrer">blog post</a></p>
</li>
<li><p>... and a <strong>whole bunch</strong> of resources in the <a href="https://github.com/jphall663/awesome-machine-learning-interpretability" rel="nofollow noreferrer">Awesome Machine Learning Interpetability</a> repo</p>
</li>
</ul>
<p><strong>NOTE</strong>: I do no longer keep this answer updated; for updates, see my answer in the AI SE thread <a href="https://ai.stackexchange.com/questions/12870/which-explainable-artificial-intelligence-techniques-are-there/24138#24138">Which explainable artificial intelligence techniques are there?</a></p>
Predictive Analytics - "why" factor & model interpretability
<p>I have the data that contains tons of x variables that are mainly categorical/nominal and my target variable is a multi-class label. I am able to build a couple models around to predict multi-class variables and compare how each of them performed. I have training and testing data. Both the training and testing data gave me good results.</p>
<p>Now, I am trying to find out &quot;why&quot; did the model predicted certain Y-variable? Meaning if I have weather data: X Variable: city, state, zip code, temp, year; Y Variable: rain, sun, cloudy, snow. I want to find out &quot;why&quot; did the model predict: rain, sun, cloudy, or snow respectfully. I used classification algorithms like multi-nominal, decision tree, ... etc</p>
<p>This may be a broad question but I need somewhere I can start researching. I can predict &quot;what&quot; but I can't see &quot;why&quot; it was predicted as rain, sun, cloudy, or snow label. Basically, I am trying to find the links between the variables that caused to predict the variable.</p>
<p>So far I thought of using correlation matrix, principal component analysis (that happened during model building process)...at least to see which are good predictors and which ones are not. Is there is a way to figure out &quot;why&quot; factor?</p>