We study the problem of explaining a rich class of behavioral properties of deep neural networks. Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent. We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.   Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.
 We study the problem of explaining a class of behavioral properties of deep neural networks, with a focus on convolutional neural networks. Examples of such properties include explaining why a network classified an input instance a particular way, why it misclassified an input, and what the essence of a class is for the network. This problem has received significant attention in recent years with the rise of deep networks and associated concerns about their opacity BID4 . This paper introduces influence-directed explanations for deep networks. It involves peering inside the network to identify neurons with high influence and then providing an interpretation for the concepts they represent. This approach enables us to interpret the inner workings of the network by drawing attention to concepts learned by the network that had a significant effect on the property that we seek to explain. In contrast to raw inputs, neurons in higher layers represent general concepts. Thus, they form a useful substrate to explain properties of interest involving many input instances, such as the essence of a class. Once influential neurons have been identified, they can be interpeted using existing techniques (e.g., visualization) to reveal the concepts they represent. Alternatively, influences can be examined directly to diagnose undesirable properties of the network.A key contribution of this paper is distributional influence, a measure for internal neurons that is axiomatically justified. Distributional influence is parameterized by a quantity of interest, a distribution of interest, and a slice of the network that allows us to reference some internal neurons in a network. It is simply the average partial derivative with respect to a neuron in a slice over the distribution of interest. This parametric measure can be appropriately instantiated to explain different properties of interest with respect to different parts of a network.Our influence measure is designed to achieve three natural desiderata: causality, distributional faithfulness, and flexibility. Capturing causality helps us identify parts of the network that when changed have the most effect on outcomes. Distributional faithfulness ensures that we evaluate the network only on points in the input distribution. This property is important since models operating on high dimensional spaces, such as neural networks, are not expected to behave reliably on instances outside the input distribution. Finally, by flexibility, we mean that the influence measure should support explanations for various properties of interest.We evaluate our approach by training convolutional neural networks on ImageNet BID8 , PubFig BID5 , and a Diabetic Retinopathy datasets. Our evaluation demonstrates that influence-directed explanations enable us to (1) characterize why inputs were classified a particular way in terms of high-level concepts represented by influential neurons (Section 3.1), (2) explain why an input was classified into a one class (e.g., sports car) rather than another (e.g., convertible) (Section 3.2), (3) demonstrate that influences localize the actual reasons used for classification better than simply examining activations (Section 3.3.1), (4) help extract the essence of what the network learned about the class (Section 3.3), and (5) assist in debugging misclassifications of a Diabetic Retinopathy classifier BID6 (Section 3.4). Influence measures are widely studied in cooperative game theory as solutions to the problem of attribution to of outcomes to participants and has applications to a wide range of settings including revenue division and voting. In this section, we highlight ideas drawn from this body of work and differences in terms of two key properties of influence measures: the marginality principle, and efficiency.The marginality principle BID14 states that an agent's attribution only depends on its own contribution to the output. Formally, this is stated as: if the partial derivatives with respect to an agent of two functions are identical throughout, then they have identical attributions for agent i. Our axiom of distributional marginality (DM) is a weaker form of this requirement that only requires equality of attribution when partial derivatives are same in the distribution.A second property, called efficiency, which is especially important for revenue division, is that attributions add up to the total value generated. This ensures that no value is left unattributed. The marginality principle, along with efficiency uniquely define the Aumann-Shapley Value BID0 . In BID12 , the Aumann-Shapley Value is used for attributions with efficiency as a justification. While it is unclear that efficiency is an essential requirement in our setting, the Aumann-Shapley value can be recovered in our framework by choosing the distribution of interest as the uniform distribution on the line segment joining an instance x and a baseline image b. Certain choices of baselines can be problematic from the point of view of distributional faithfulness, since the line segment of linear combinations between them might lie significantly out of distribution. The particular baseline chosen in BID12 is the zero vector, where the line segment represents scaled images, and could be reasonably called within distribution.
We present an influence-directed approach to constructing explanations for the behavior of deep convolutional networks, and show how it can be used to answer a broad set of questions that could not be addressed by prior work. A way to measure influence that satisfies certain axioms, and a notion of influence that can be used to identify what input part is most influential for the output of a neuron in a deep neural network. This paper proposes to measure the influence of single neurons with regard to a quantity of interest represented by another neuron.