Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models. We propose Mahé, a novel approach to provide Model-Agnostic Hierarchical Explanations of how powerful machine learning models, such as deep neural networks, capture these interactions as either dependent on or free of the context of data instances. Specifically, Mahé provides context-dependent explanations by a novel local interpretation algorithm that effectively captures any-order interactions, and obtains context-free explanations through generalizing context-dependent interactions to explain global behaviors. Experimental results show that Mahé obtains improved local interaction interpretations over state-of-the-art methods and successfully provides explanations of interactions that are context-free. State-of-the-art machine learning models, such as deep neural networks, are exceptional at modeling complex dependencies in structured data, such as text BID44 BID40 , images BID6 BID12 , and DNA sequences BID0 BID48 . However, there has been no clear explanation on what type of dependencies are captured in the black-box models that perform so well BID30 .In this paper, we make one of the first attempts at solving this important problem through interpreting two forms of structures, i.e., context-dependent representations and context-free representations. A context-dependent representation is the one in which a model's prediction depends specifically on a data instance level (such as a sentence or an image). In order to illustrate the concept, we consider an example in image analysis. A yellow round-shape object can be identified as the sun or the moon given its context, either bright blue sky or dark night. A context-free representation is one where the representation behaves similarly independent of instances (i.e., global behaviors). In a hypothetical task of classifying sentiment in sentences, each sentence carries very different meaning, but when "not" and "bad" depend on each other, their sentiment contribution is almost always positive -i.e., the structure is context-free.To investigate context-dependent and context-free structure, we lend to existing definitions in interpretable machine learning BID29 BID13 . A context-dependent interpretation is a local interpretation of the dependencies at or within the vicinity of a single data instance. Conversely , a context-free interpretation is a global interpretation of how those dependencies behave in a model irrespective of data instances. In this work , we study a key form of dependency: an interaction relationship between the prediction and input features. Interactions can describe arbitrarily complex relationships between these variables and are commonly captured by state-of-the-art models like deep neural networks BID42 . Interactions which are context-dependent or context-free are therefore local or global interactions, respectively.We propose Mahé, a framework for explaining the context-dependent and context-free structures of any complex prediction model, with a focus on explaining neural networks. The context-dependent explanations are built based on recent work on local intepretations (such as BID29 ). Specifically, Mahé takes as input a model to explain and a data instance, and returns a hierarchical explanation, a format proposed by to show local group-variable relationships used in predictions (Figure 1 ). To provide context-free Input into complex ML model:Step FORMULA0 Step FORMULA3 Step 3 Fit ( ) In this work, we proposed Mahé, a model-agnostic framework of providing context-dependent and context-free explanations of local interactions. Mahé has demonstrated the capability of outperforming existing approaches to local interaction interpretation and has shown that local interactions can be context-free. In future work, we wish to make the process of finding context-free interactions more efficient, and study to what extent model behavior can be changed by editing its interactions or univariate effects. Finally, we would like to study the interpretations provided by Mahé more closely to find new insights into structured data. Table 6 : Examples of context-dependent hierarchical explanations on Sentiment-LSTM. The interaction attribution of g K (·) is shown at each K − 1 level, K ≥ 1 ( §4.1) in color. Green means positively contributing to sentiment, and red the opposite. Visualized attributions of linear LIME and Mahé are normalized to the max attribution magnitudes (max magn.) shown. Top-5 attributions by magnitude are shown for LIME.
A new framework for context-dependent and context-free explanations of predictions The authors extend the linear local attribution method LIME for interpreting black box models, and propose a method to discern between context-dependent and context-free interactions. A method that can provide hierarchical explanations for a model, including both context-dependent and context-free explanations by a local interpretation algorithm.