The interpretability of neural networks has become crucial for their applications in real world with respect to the reliability and trustworthiness. Existing explanation generation methods usually provide important features by scoring their individual contributions to the model prediction and ignore the interactions between features, which eventually provide a bag-of-words representation as explanation. In natural language processing, this type of explanations is challenging for human user to understand the meaning of an explanation and draw the connection between explanation and model prediction, especially for long texts. In this work, we focus on detecting the interactions between features, and propose a novel approach to build a hierarchy of explanations based on feature interactions. The proposed method is evaluated with three neural classifiers, LSTM, CNN, and BERT, on two benchmark text classification datasets. The generated explanations are assessed by both automatic evaluation measurements and human evaluators. Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models, and understandable to humans. Deep neural networks have become a significant component in natural language processing (NLP), achieving state-of-the-art performance in various NLP tasks, such as text classification (Kim, 2014) , question answering (Rajpurkar et al., 2016) , and machine translation (Bahdanau et al., 2014) . However, the lack of understanding on their decision making leads them to be characterized as black-box models and increases the risk of applying them in real-world applications (Lipton, 2016) . Producing interpretable decisions has been a critical factor on whether people will trust and use the neural network models (Ribeiro et al., 2016) . Most of existing work on local explanation generation for NLP focuses on producing word-level explanations (Ribeiro et al., 2016; Lei et al., 2016; Plumb et al., 2018) , where a local explanation consists of a set of words extracted from the original text. Figure 1 presents an example sentence with its sentiment prediction and corresponding word-level explanation generated by LIME (Ribeiro et al., 2016) . Although the LIME explanation captures a negative sentiment word waste, it presents the explanation in a bag-of-words format. Without resorting to the original text, it is difficult for us to understand the contribution of word a and of, as both of them have no sentiment polarity. The situation will become even more serious when this type of explanations are extracted from longer texts. In this work, we present a novel method to construct hierarchical explanations of a model prediction by capturing the interaction between features. Ultimately, our method is able to produce a hierarchical structure as illustrated in Figure 1 . Produced by the proposed method, this example provides a comprehensive picture of how different granularity of features interacting with each other for model prediction. With the hierarchical structure, this example tells us how the words and phrases are combined and what are the contributions of words and phrases to the final prediction. For example, the contribution of the phrase of good is dominated by the word waste, which eventually leads to the right prediction. Figure 1: A NEGATIVE movie review a waste of good performance with a LIME explanation and a hierarchical explanation, where the color of each block represents the importance of the corresponding word/phrase with respect to the model prediction. To capture feature interactions, we adopt the interacted Shapley value (Lundberg et al., 2018) , an extension of Shapley value (Shapley, 1953) from cooperative game theory, to measure the interactions between features. Based on the interaction scores, we propose a top-down method, called INTERSHAPLEY, to segment a text recursively into phrases and then words eventually. The proposed method is evaluated on text classification tasks with three typical neural network models: long short term memory networks (Hochreiter & Schmidhuber, 1997, LSTM) and convolutional neural networks (Kim, 2014, CNN) , and a state-of-the-art model BERT (Devlin et al., 2018) on some benchmark datasets. The comparison of our method is against several competitive baselines from prior work on explanation generation, including Leave-one-out (Li et al., 2016) , Contextual Decomposition (CD) (Murdoch et al., 2018) and its hierarchical extension (ACD) (Singh et al., 2019) , L-and C-Shapley (Chen et al., 2018) , and LIME (Ribeiro et al., 2016) . Our contribution of this work is three-fold: (1) we propose an effective method to calculate feature importance and extend the Shapley value to measure feature interactions; (2) we design a top-down segmentation algorithm to build hierarchical interpretations based on feature interactions; (3) we compare the proposed method with several competitive baselines via both automatic and human evaluations, and show the INTERSHAPLEY method outperforms the existing methods on both wordand phrase-level explanations.
A novel approach to construct hierarchical explanations for text classification by detecting feature interactions. A novel method for providing explanations for predicitions made by text classifiers that outperforms baselines on word level importance scores, and a new metric, cohesion loss, to evaluate span-level importance. An interpretation method based on feature interactions and feature importance score as compared to independent feature contributions.