Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample. These explanations are appealing due to their simplicity and local fidelity. However, they do not provide information about the general behavior of the model. We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions. These global explanations take the form of feature shapes, which are more expressive than feature attributions. Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets. A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc Recent research in interpretability has focused on developing local explanations: given an existing model and a sample, explain why the model made a particular prediction for that sample BID40 . The accuracy and quality of these explanations have rapidly improved, and they are becoming important tools to understand model decisions for individual samples. However, the human cost of examining multiple local explanations can be prohibitive with today's large data sets, and it is unclear whether multiple local explanations can be aggregated without contradicting each other BID41 BID0 .In this paper, we are interested in global explanations that describe the overall behavior of a model. While usually not as accurate as local explanations on individual samples, global explanations provide a different, complementary view of the model. They allow us to clearly visualize trends in feature space, which is useful for key tasks such as understanding which features are important, detecting unexpected patterns in the training data and debugging errors learned by the model.We propose to use model distillation techniques BID7 BID24 to learn global additive explanations of the form DISPLAYFORM0 to approximate the prediction function of the model F (x). Figure 1 illustrates our approach. The output of our approach is a set of p feature shapes {h i } p 1 that can be composed to form an explanation model that can be quantitatively evaluated. Through controlled experiments, we empirically validate that feature shapes provide accurate and interesting insights into the behavior of complex models. In this paper, we focus on interpreting F from fully-connected neural nets trained on tabular data.Our goal is not to replace local explanations nor to explain how the model functions internally. What we claim is that we can complement local explanations with global additive explanations that clearly illustrate the relationship between input features and model predictions. Our contributions are:• We propose to learn global additive explanations for complex, non-linear models such as neural nets.• We leverage powerful generalized additive models in a model distillation setting to learn feature shapes that are more expressive than feature attributions Figure 1 : Given a black box model and unlabeled samples (new unlabeled data or training data with labels discarded), our approach leverages model distillation to learn feature shapes that describe the relationship between input features and model predictions.• We perform a quantitative comparison of feature shapes to other global explanation methods in terms of fidelity to the model being explained, accuracy on independent test data, and interpretability through a user study. We presented a method for "opening up" complex models such as neural nets trained on tabular data. The method, based on distillation with high-accuracy additive models, has clear advantages over other approaches that learn additive explanations but not using distillation, and non-additive explanations using distillation. Our global additive explanations do not aim to compete with local explanations or non-additive explanations such as decision trees. Instead, we show that different interpretable representations work well for different tasks, and global additive explanations are valuable for important tasks that require quick understanding of feature-prediction relationships. Although in this paper we focus on explaining FNNs, the method will work with any classification or regression model including random forests and CNNs, but is not designed to work with raw image inputs such as pixels where providing a global explanation in terms of input pixels is not meaningful. One way to address this is to define more meaningful "features", e. hi (xi) hi (xi) hi(xi) Figure A1 : Feature shapes for features x 1 to x 9 of F 1 from Section 4.1. Notice how x 9 , which is a noise feature that does not affect F 1 , has been assigned an importance of approximately 0 throughout its range. The feature shape of x 10 , another noise feature, is very similar to x 9 and hence not included here.hi (xi) hi (xi) hi(xi) FIG0 : Feature shapes for features x 1 to x 9 of F 2 from Section 4.1. Notice how x 9 , which is a noise feature that does not affect F 2 , has been assigned an importance of approximately 0 throughout its range. The feature shape of x 10 , another noise feature, is very similar to x 9 and hence not included here. Table A1 : Accuracy and fidelity of global explanation models across 1H and 2H teacher neural nets and datasets. TAB4 is a subset of this table with only 2H neural nets.In general, the lower-capacity 1H neural nets are easier to approximate (i.e. better student-teacher fidelity), but their explanations are less accurate on independent test data. Students of simpler teachers tend to be less accurate even if they are faithful to their (simple) teachers. One exception is the FICO data, where the fidelity of the 2H explanations is better. Our interpretation is that many features in the FICO data have almost linear feature shapes (see Figure A5 for a sample of features), and the 2H model may be able to better capture fine details while being simple enough that it can still be faithfully approximated. The accuracy of the SAT and SAS for 1H and 2H neural nets are comparable, taking into account the confidence intervals.On the Magic data, the fidelity of the gGRAD explanation to the 1H neural net (see * in Table A1 ) is markedly worse than other explanation methods. We investigate the individual gradients of the 1H neural net with respect to each feature ( DISPLAYFORM0 ∂xi in GRAD equation in Section 3). 99% of them have reasonable values (between -5.6 and 6). However, 3 are larger than 1,000 (with none between 6 and 1,000) and 13 are lower than -1,000 (with none between -1,000 and -5.6), resulting in the ensuing gGRAD explanation generating extreme predictions for several samples that are not faithful to the teacher's predictions. Because AUC is a ranking loss, accuracy (AUC) is less affected than fidelity (RMSE) by the presence of these extreme values. This shows that gGRAD explanations may be problematic when individual gradients are arbitrarily large, e.g. in overfitted neural nets. Figure A7 , removing the color and number of samples in each node, to improve readability for the user study.
We propose to leverage model distillation to learn global additive explanations in the form of feature shapes (that are more expressive than feature attributions) for models such as neural nets trained on tabular data. This paper incorporates Generalized Additive Models (GAMs) with model distillation to provide global explanations of neural nets.