Does a Neural Network Really Encode Symbolic Concepts?
Recently, a series of studies have tried to extract interactions between
input variables modeled by a DNN and define such interactions as concepts
encoded by the DNN. However, strictly speaking, there still lacks a solid
guarantee whether such interactions indeed represent meaningful concepts.
Therefore, in this paper, we examine the trustworthiness of interaction
concepts from four perspectives. Extensive empirical studies have verified that
a well-trained DNN usually encodes sparse, transferable, and discriminative
concepts, which is partially aligned with human intuition.
Mingjie Li, Quanshi Zhang
# Does a Neural Network Really Encode Symbolic Concept?

###### Abstract

Recently, a series of studies have tried to extract interactions between input variables modeled by a DNN and define such interactions as concepts encoded by the DNN. However, strictly speaking, there still lacks a solid guarantee whether such interactions indeed represent meaningful concepts. Therefore, in this paper, we examine the trustworthiness of interaction concepts from four perspectives. Extensive empirical studies have verified that a well-trained DNN usually encodes sparse, transferable, and discriminative concepts, which is partially aligned with human intuition. _The code will be released when the paper is accepted._

## 1 Introduction

Understanding the black-box representation of deep neural networks (DNNs) has received increasing attention in recent years. Unlike graphical models with interpretable internal logic, the layerwise feature processing in DNNs makes it naturally difficult to explain DNNs from the perspective of symbolic concepts. Instead, previous studies interpreted DNNs from other perspectives, such as illustrating the visual appearance that maximizes the inference score [20, 17], and estimating attribution/importance/saliency of input variables [14, 15, 16]. Zhou et al. [21], Bau et al. [20], Kim et al. [20] visualized the potential correspondence between convolutional filters in a DNN and visual concepts in an empirical manner.

**Unlike previous studies, a series of studies [13, 20, 21, 22] tried to define and propose an exact formulation for the concepts encoded by a DNN.** Specifically, these studies discovered that a well-trained DNN usually encoded various interactions between different input variables, and the inference score on a specific input sample could be explained by numerical effects of different interactions. Thus, they claimed that each interaction pattern was a symbolic concept encoded by the DNN.

Specifically, let us consider a DNN given a sample with \(n\) input variables \(N=\{1,2,...,n\}\)_e.g._, given a sentence with five words "_he is a green hand._" The DNN usually does not use each individual input variable for inference independently. Instead, the DNN lets different input variables interact with each other to construct concepts for inference. For example, a DNN may memorize the interaction between words in \(S=\{\textit{green, hand}\}\) with a specific numerical contribution \(I(S)\) to push the DNN's inference towards the meaning of a "_beginmer._" Such a combination of words is termed an _interaction concept_. Each interaction concept \(S\subseteq N\) represents the AND relationship between input variables in \(S\). Only the co-appearance of input variables in \(S\) can make an interaction effect \(I(S)\) on the network output. In contrast, masking any word in \(\{\textit{green, hand}\}\) removes the interaction effect \(I(S)\). In this way, Ren et al. [20] proved that the inference score \(y\) of a trained DNN on each sample can be written as the sum of effects of all potential symbolic concepts \(S\subseteq N\), _i.e._\(y=\sum_{S\subseteq N}I(S)\).

_However, the claim that "a DNN encodes symbolic concepts" is too counter-intuitive. Ren et al. [20] just formulated \(I(S)\) that satisfied \(y=\sum_{S\subseteq N}I(S)\). Current studies have not provided sufficient support for the claim that a DNN really learns symbolic concepts. In fact, we should not ignore another potential situation that the defined effect \(I(S)\) is just a mathematical transformation that ensures the decomposition of network output into concepts \(y=\sum_{S\subseteq N}I(S)\), rather than faithfully representing a meaningful and transferable concept learned by a DNN._

Therefore, in this paper, we aim to give a quantitative verification of the concept-emerging phenomenon, _i.e._, whether a well-trained DNN summarizes transferable symbolic knowledge from chaotic raw data, like what human brain does. Or the defined effect \(I(S)\) is just a mathematical game without clear meanings. To this end, we believe that if a well-trained DNN really encodes certain concepts, then the concepts are supposed to satisfy the following four requirements.

* **Representing network inference using a few concepts.** If a DNN really learns symbolic concepts, then the DNN's inference on a specific sample is supposed to be concisely explained by a small number of salient concepts, rather than a large number of concepts, according to both
Occam's Razor and people's intuitive understanding towards concepts. In fact, the sparsity of concepts in a specific sample has been discussed by Ren et al. (2021). In this paper, we further conduct extensive experiments on more diverse DNNs, in order to verify that a well-trained DNN usually extracts sparse concepts from each sample for inference.

\(\bullet\)**A transferable concept dictionary through different samples.** If a DNN can use a relatively small set \(\mathbf{D}\) of salient concepts, namely a concept dictionary, to approximate inference scores on different samples in a category, \(\forall\mathbf{x},y\approx\sum_{S\in\mathbf{D}}I(S|\mathbf{x})\), then we consider the concept dictionary \(\mathbf{D}\) represents common features shared by different samples in the category. Otherwise, if Ren et al. (2021) extract a fully different set of concepts from each different sample in the same category, then these concepts probably represent noisy signals. In other words, convincing concepts must be stably extracted with high transferability through different samples in the same category.

\(\bullet\)**Transferability of concepts across different DNNs.** Similarly, when we train different DNNs for the same task, different DNNs probably learn similar sets of concepts if they really memorize the defined "concepts" as basic inference patterns for the task.

\(\bullet\)**Discrimination power of concepts.** Furthermore, if a DNN learns meaningful concepts, then these concepts are supposed to exhibit certain discrimination power in the classification task. The same concept extracted from different samples needs to consistently push the DNN towards the classification of a certain category.

To this end, we conducted experiments on various DNNs trained on different datasets for classification tasks, including tabular datasets, image datasets, and point-cloud datasets. We found that all these trained DNNs encoded transferable concepts. On the other hand, we also investigated a few extreme cases, in which the DNN either collapsed into simple linear models or failed to learn transferable and discriminative concepts. In sum, although we cannot theoretically prove the phenomenon of the emergence of transferable concepts, this phenomenon indeed happened for most tasks in our experiments.

**Contributions** of this paper can be summarized as follows. (1) Besides the sparsity of concepts, we propose three more perspectives to examine the concept-emerging phenomenon of a DNN, _i.e._, whether the DNN summarizes transferable symbolic knowledge from chaotic raw data. (2) Extensive empirical studies on various tasks have verified that a well-trained DNN usually encodes transferable interaction concepts. (3) We also discussed three extreme cases, in which a DNN is unlikely to learn transferable concepts.

## 2 Related works

**Understanding the black-box representation of DNNs.** Many explanation methods have been proposed to explain DNNs from different perspectives. Typical explanation methods include visualizing patterns encoded by a DNN (Simonyan et al., 2013; Zeiler and Fergus, 2014; Yosinski et al., 2015; Dosovitskiy and Brox, 2016), estimating the attribution/importance/saliency of each input variable (Ribeiro et al., 2016; Sundararajan et al., 2017; Lundberg and Lee, 2017; Fong and Vedaldi, 2017; Zhou et al., 2016; Selvaraju et al., 2017), and learning feature vectors potentially correspond to semantic concepts (Kim et al., 2018). Some studies explained a DNN by distilling the network into another interpretable model (Frosst and Hinton, 2017; Che et al., 2016; Wu et al., 2018; Zhang et al., 2018; Vaughan et al., 2018; Tan et al., 2018). However, most explanation methods did not try to disentangle concepts encoded by a DNN.

**Interaction encoded by DNNs.** Many recent studies have focused on quantifying interactions between input variables learned by a DNN (Sorokina et al., 2008; Murdoch et al., 2018; Singh et al., 2018; Jin et al., 2019; Janizek et al., 2020). In game theory, Grabisch and Roubens (1999); Sundararajan et al. (2020); Tsai et al. (2022) proposed in

Figure 1: Visualization of interaction concepts \(S\) extracted by PointNet on different samples in the ShapeNet dataset. The histograms show the distribution of interaction effects \(I(S|\mathbf{x})\) over samples in the “motorbike” category, where \(S\) is extracted as a salient concept.
teraction metrics from different perspectives. Recently, Ren et al. (2021) defined concepts encoded by a DNN as the interaction between input variables. Ren et al. (2023) further optimized baseline values of input variables based on such concepts. Besides, Deng et al. (2022) found that DNNs were less likely to encode interaction concepts of intermediate complexity. However, strictly speaking, there still lacks theoretical guarantee for the interaction to really represent concepts. Therefore, in this paper, we examine the trustworthiness of the interaction concepts from four perspectives.

## 3 Emergence of transferable concepts

### Preliminaries: representing network inferences using interaction concepts

It is widely believed that the learning of a DNN can be considered as a regression problem, instead of explicitly learning symbolic concepts like how graphical models do. However, a series of studies (Ren et al., 2021, 2023; Deng et al., 2022) have discovered that given a sufficiently-trained DNN for a classification task, its inference logic on a certain sample can usually be rewritten as the detection of specific concepts. In other words, the DNN's inference score on a specific sample can be sparsely disentangled into effects of a few concepts.

**Disentangling the DNN's output as effects of interaction concepts.** Specifically, let us consider a trained DNN \(v:\mathbb{R}^{n}\rightarrow\mathbb{R}\) and an input sample \(\mathbf{x}\) with \(n\) input variables indexed by \(N=\{1,2,...,n\}\). Here, we assume the network output \(v(\mathbf{x})\in\mathbb{R}\) is a scalar. Note that different settings can be applied to \(v(\mathbf{x})\). For example, for multi-category classification tasks, we may set \(v(\mathbf{x})=\log\frac{p(y=y^{\text{run}}|\mathbf{x})}{1-p(y=y^{\text{run}}|\mathbf{x})}\in \mathbb{R}\) by following (Deng et al., 2022). Then, given a function \(v(\mathbf{x})\), Ren et al. (2021) have proposed the following metric to quantify the interaction concept that is comprised of input variables in \(S\subseteq N\).

\[I(S|\mathbf{x})\triangleq\sum\nolimits_{T\subseteq S}(-1)^{|S|-|T|}\cdot v(\mathbf{x} _{T}) \tag{1}\]

where \(\mathbf{x}_{T}\) denotes the input sample when we keep variables in \(T\subseteq N\) unchanged and mask variables in \(N\backslash T\) using baseline values1.

Footnote 1: For all tabular datasets and the image datasets (the _CelebA-eyeglasses_ dataset and the _CUB-binary_ dataset), the baseline value of each input variable was set as the mean value of this variable over all samples (Dabkowski and Gal, 2017). For grayscale digital images in the _MNIST-3_ dataset, the baseline value of each pixel was set as zero (Ancona et al., 2019). For the point-cloud dataset, the baseline value was set as the center of the entire point cloud (Shen et al., 2021).

**Here, the interaction concept \(I(S|\mathbf{x})\) extracted from

Figure 2: Visualization of interaction concepts \(S\) extracted by two _MLP-5_ networks3, which are trained on (a) the _wifi_ dataset3 and (b) the _tic-lac-toe_ dataset3. The histograms show (a) the distribution of interaction effects \(I(S|\mathbf{x})\) over samples in the \(4^{\text{th}}\) category, and (b) the distribution of interaction effects \(I(S|\mathbf{x})\) over samples in sub-categories4 with patterns \(x_{4}=x_{5}=x_{6}=1\) and \(x_{3}=x_{6}=x_{9}=1\).
the input \(\mathbf{x}\) encodes an AND relationship (interaction) between input variables in \(S\). For example, let us consider three image regions of \(S=\{\textit{eyes, nose, mouth}\}\) that form the _"face"_ concept in a face classification task. Then, \(I(S|\mathbf{x})\) measures the numerical effect of the concept on the classification score \(v(\mathbf{x})\). Only when all image regions of _"eyes"_, _"nose"_, and _"mouth"_ co-appear in the input image, the _"face"_ concept is activated, and contributes a numerical effect \(I(S|\mathbf{x})\) to the classification score. Otherwise, if any region is masked, then the _"face"_ concept cannot be formed, which removes the interaction effect, making \(I(S|\mathbf{x}_{\text{masked}})=0\).

Mathematically, the above definition for an interaction concept can be understood as the Harsanyi dividend [17] of \(S\)_w.r.t._ the DNN \(v\). It has been proven that the DNN's inference score \(v(\mathbf{x})\) can be disentangled into the sum of effects of all potential concepts, as follows.

\[v(\mathbf{x})=\sum\nolimits_{S\subseteq N}I(S|\mathbf{x})\ \approx\sum\nolimits_{S\in \Omega_{\mathbf{x}}}I(S|\mathbf{x}) \tag{2}\]

In particular, all interaction concepts can be further categorized into a set of **salient concepts**\(S\in\Omega_{\mathbf{x}}\) with considerable effects \(I(S|\mathbf{x})\), and a set of ignorable **noisy patterns** with almost zero effects \(I(S|\mathbf{x})\approx 0\).

Note that the Harsanyi dividend \(I(S|\mathbf{x})\) also satisfies many desirable axioms/theorems, as introduced in Appendix A. The interaction effects can be further optimized using various tricks in [14] to pursue much more sparsity.

### Visualization of interaction concepts

In this section, we visualize interaction concepts extracted from point-cloud data and tabular data. Note that a sample in the ShapeNet dataset [20] usually contains 2500 3D points. To simplify the visualization, we simply consider 8-10 semantic parts on the point cloud \(\mathbf{x}\), which has been provided by the dataset.2 Each semantic part is taken as a "single" input variable to the DNN. In this way, we visualize concepts consisting of semantic parts.

Footnote 2: For example, the ShapeNet dataset has provided the annotated parts for the _motorbike_ category, including _gas tank_, _seat_, _handle_, _light_, _front wheel_, _back wheel_, _front frame_, _mid frame_, and _back frame_. Please see Appendix B.2 for details on the annotation of semantic parts.

Fig. 1 shows interaction concepts \(S\) and the corresponding effects \(I(S|\mathbf{x})\) extracted by PointNet [20] from different samples \(\mathbf{x}\) in the ShapeNet dataset. We find that the interaction concept \(S=\{\textit{light, front wheel, mid frame}\}\) on five samples all makes positive effects \(I(S|\mathbf{x})>0\) to the PointNet's output, whereas the interaction concept \(S\ =\ \{\textit{handle, front wheel, front frame}\}\) usually makes negative effects \(I(S|\mathbf{x})<0\) to the PointNet's output. Similarly, Fig. 2 shows interaction concepts extracted from two tabular datasets, _i.e._, the _wifi_ dataset3, and the _tic-tac-toe_ dataset3. We visualize interactions between different _received signal strength indicators_ (RSSIs) in the _wifi_ dataset, and interactions between _board states_ in the _tic-tac-toe_ dataset. We also find that the same interaction concept usually makes similar effects to the network output on different input samples, which supports the conclusion in Section 3.3.2.

Footnote 3: Please see the _experimental settings_ paragraph at the end of Section 3 for details on datasets and DNNs.

### Does a DNN really learn symbolic concepts?

Although Ren et al. [2021] have claimed that the metric \(I(S|\mathbf{x})\) in Eq. (1) quantifies symbolic concepts encoded by a DNN, there is still no theory to guarantee a subset \(S\subseteq N\) with a salient effect \(I(S|\mathbf{x})\) faithfully represents a meaningful and transferable concept. Instead, we should not ignore the possibility that \(I(S|\mathbf{x})\) is just a mathematical transformation that ensures \(v(\mathbf{x})=\sum_{S\subseteq N}I(S|\mathbf{x})\) in Eq. (2) on each specific sample. Therefore, in this study, we examine the counter-intuitive conjecture that a DNN learns symbolic concepts from the following four perspectives.

#### 3.3.1 Sparsity of the encoded concepts

According to Eq. (2), the DNN may encode at most \(2^{n}\) symbolic concepts in \(2^{N}\triangleq\{S:S\subseteq N\}\)_w.r.t._ the \(2^{n}\) different combinations of input variables. **However, a distinctive property of symbolism, which is different from connectionism, is that people usually would like to use _a small number of explicit_ symbolic concepts to represent the knowledge, instead of using _extensive fuzzy_ features.**

Thus, we hope to examine whether a DNN's inference score \(v(\mathbf{x})\) on a specific sample can be summarized into effects of a small number of salient concepts \(v(\mathbf{x})\approx\sum_{S\in\Omega_{\mathbf{x}}}I(S|\mathbf{x})\), rather than using an exponential number of concepts _w.r.t._ all subsets \(S\subseteq N\). To be precise, a faithful conceptual representation requires most concepts \(S\subseteq N\) to be noisy patterns with negligible effects \(I(S|\mathbf{x})\approx 0\). Only a few salient concepts \(S\) in \(\Omega_{\mathbf{x}}\) make considerable effects \(I(S|\mathbf{x})\).

To this end, Ren et al. [2021] have made a preliminary attempt to explain a DNN's inference score \(v(\mathbf{x})\) on a specific sample \(\mathbf{x}\) as interaction effects \(I(S|\mathbf{x})\) of a small number of concepts. Specifically, they used a few top-ranked salient interaction concepts to explain the inference score of LSTMs [10] and CNNs [19] trained for sentiment classification and linguistic acceptance classification tasks on the SST-2 dataset [21] and the CoLA dataset [20].

**Experiments.** In this paper, we further examined whether most DNNs, which were trained for much more diverse tasks on different datasets, all encoded very sparse salient concepts. To this end, we trained various DNNs3 on tabular datasets (the _tic-tac-toe_ dataset3 and the _wifi_ dataset3), image datasets
(the _MNIST-3_ dataset3 and the _CelebA-eyeglasses_ dataset3), and a point-cloud dataset (the _ShapeNet_ dataset3). The interaction effects can be further optimized using various tricks in [11] to pursue much more sparsity. Fig. 3 shows the normalized interaction strength of different concepts \(|\tilde{I}(S|\mathbf{x})|\triangleq|I(S|\mathbf{x})|/\max_{S^{\prime}}|I(S^{\prime}| \mathbf{x})|\) in a descending order for each DNN. Each curve shows the strength averaged over different samples in the dataset. We found that most concepts had little effects on the output \(|I(S|\mathbf{x})|\approx 0\), which verified the sparsity of the encoded concepts.

Footnote 3: In this paper, when we needed to analyze of samples in a specific category, we used positive samples in the _MNIST-3_, _CelebA-eyeglasses_, and _CUB-binary_ datasets, samples in the 4th category in the _wifi dataset_, and samples in the “motorbike” category in the ShapeNet dataset. For the _ic-tac-tac-toe_ dataset, since there exists eight sub-categories among positive samples, we used samples in the sub-category with the pattern \(x_{4}=x_{5}=x_{6}=1\).

**Salient concepts.** According to the above experiments, we can define the set of salient concepts as \(\Omega_{\mathbf{x}}=\{S:|I(S|\mathbf{x})|>\tau\}\), subject to \(\tau=0.05\cdot\max_{S}|I(S|\mathbf{x})|\). As Fig. 3 shows, there were only about 20-80 salient concepts extracted from an input sample, and all other concepts have ignorable effects on the network output.

#### 3.3.2 Transferability over different samples

Beyond sparsity, the transferability of concepts is more important. If \(I(S|\mathbf{x})\) is just a tricky mathematical transformation on \(v(\mathbf{x}_{S})\) without representing meaningful concepts, then each salient concept \(I(S|\mathbf{x})\) extracted from the input sample \(\mathbf{x}\) probably cannot be transferred to another input sample, _i.e.,_ we cannot extract the same salient concept consisting of variables in \(S\) in the second sample, due to sparsity of salient concepts.

Therefore, in this section, **we aim to verify whether concepts extracted from a sample can be transferred to other samples.** This task is actually equivalent to checking whether there exists a common concept dictionary, which contains most salient concepts extracted from different samples in the same category.

Given a well-trained DNN, we construct a relatively small dictionary \(\mathbf{D}_{k}\subseteq 2^{N}\) containing the top-\(k\) frequent concepts in different samples, and check whether such a dictionary contains most salient concepts in \(\Omega_{\mathbf{x}}\) extracted from each sample \(\mathbf{x}\). The concept dictionary \(\mathbf{D}_{k}\) is constructed based on a greedy strategy. Specifically, we first extract a set of salient concepts \(\Omega_{\mathbf{x}}\) from each input sample \(\mathbf{x}\). Then, we compute the frequency of each concept \(S\) being a salient concept over different samples. Finally, the concept dictionary \(\mathbf{D}_{k}\) is constructed to contain the top-\(k\) interaction concepts with the highest frequencies.

Then, we design the metric \(\rho(k)\triangleq\mathbb{E}_{\mathbf{x}}[|\mathbf{D}_{k}\cap\Omega_{\mathbf{x}}|/| \Omega_{\mathbf{x}}|]\) to evaluate the average ratio of concepts extracted from an input sample that is covered by the concept dictionary \(\mathbf{D}_{k}\). Theoretically, if we construct a larger dictionary \(\mathbf{D}_{k}\) with more concepts (a larger \(k\) value), then the dictionary can explain more concepts.

**Experiments.** We conducted experiments to show whether there existed a small concept dictionary that could explain most concepts encoded by the DNN. Specifically, we constructed a concept dictionary to explain samples in a certain category in each dataset4. In this experiment, we temporarily extracted salient concepts using \(\tau=0.1\cdot\max_{S}|I(S|\mathbf{x})|\) to construct \(\Omega_{\mathbf{x}}\)5. Fig. 4 shows the increase of the average explanation ratio \(\rho(k)\) along with the

Figure 4: The change of the average explanation ratio \(\rho(k)\) along with the size \(k\) of the concept dictionary \(\mathbf{D}_{k}\).

Figure 3: Normalized strength of interaction effects of different concepts in a descending order. DNNs trained for different tasks all encode sparse salient concepts.
increasing size \(k\) of the concept dictionary \(\mathbf{D}_{k}\). We found that there usually existed a concept dictionary consisting of 30-100 concepts, which could explain more than 60%-80% salient concepts encoded by the DNN. Besides, Fig. 1(right) and Fig. 2(right) also show histograms of effects \(I(S|\mathbf{x})\) over different samples4, where \(S\) was extracted as a salient concept. We found that the same interaction concept usually made similar effects on different samples. This verified that the DNN learned transferable concepts over different samples.

Footnote 4: We note that the
a salient and negative effect on the classification score. In this way, the discrimination power of a salient concept \(S\) can be measured as \(\beta(S)=\max(m_{S}^{+},m_{S}^{-})/(m_{S}^{+}+m_{S}^{-})\). A larger value of \(\beta(S)\) indicates a higher discrimination power of the concept \(S\).

**Experiments.** Note that different concepts are of different importance in the classification of a category. Some concepts frequently appear in different samples and make salient effects, while other concepts only appear in very few concepts. Therefore, we use the frequency of the concept \(\alpha(S)\) as a weight to compute the average discrimination power of all concepts, which is given as \(\bar{\beta}\triangleq\sum_{S}[\alpha(S)\cdot\beta(S)]/\sum_{S}[\alpha(S)]\). The frequency of a concept is defined as \(\alpha(S)\triangleq(m_{S}^{+}+m_{S}^{-})/m\). The selection of datasets and the training of DNNs are introduced in the _experimental settings_ paragraph at the end of Section 3. Fig. 6 shows the average discrimination power of concepts in different frequency intervals. We found that the average discrimination power \(\bar{\beta}\) of concepts was usually higher than \(0.8\), which verified the discrimination power of extracted concepts.

### When DNNs do not learn transferable concepts

It is worth noting that all the above work just conducts experiments to show the concept-emerging phenomenon in different DNNs for different tasks. We do not provide, or there may even do not exist, a theoretical proof for such a concept-emerging phenomenon, although the concept-emerging phenomenon **does exist** in DNNs for most applications. Therefore, in this subsection, we would like to discuss the following three extreme cases, in which a DNN does not learn symbolic concepts.

In the three extreme cases, DNNs may either collapse to simple models close to linear regressions, or learn non-transferable indiscriminative concepts, although the network output can still be represented as the sum of interaction effects of these concepts.

\(\bullet\)**Case 1: When there exists label noise.** If the ground-truth label for classification is incorrectly annotated on some samples, then the DNN usually has to memorize each incorrectly-labeled training sample for classification without summarizing many common features from such chaotic annotations. Thus, in this case, the DNN usually encodes more non-transferable concepts.

_Experimental verification._ In this experiment, we constructed datasets with noisy labels to check whether DNNs trained on these datasets did not learn transferable concepts with high discrimination power. To this end, given a clean dataset, we first selected and randomly labeled a certain portion \(r\) of training samples in the dataset, so as to train a DNN. Specifically, we constructed a series of datasets by assigning different ratios \(r\) of samples with incorrect labels. We used the _wifi_ dataset[3] to construct new datasets by adding different ratios \(r\) of noisy labels. Then, we trained an MLP-5 network[3] on each of these datasets. We examined the transferability and discrimination power of the extracted concepts[4] on each MLP-5 network. We extracted concept dictionaries of different sizes based on each MLP-5 network (please see Section 3.3.2 for details). Fig. 7(left) shows that if there was significant label noise in the dataset, the concept dictionary usually explained much fewer concepts encoded by the network, which indicated low transferability of the learned concepts. Besides, Fig. 7(right) shows the average discrimination power \(\bar{\beta}\) of the extracted concepts also decreased when we assigned more training samples with random labels. This verified that the DNN usually could not learn transferable and discriminative concepts from samples that were incorrectly labeled.

\(\bullet\)**Case 2: When input samples are noisy.** In fact, this case can be extended to a more general scenario, _i.e._, when the task is difficult to learn, there is no essential difference between the difficult data and noisy data for the DNN. Specifically, when training samples are noisy and lack meaningful

Figure 6: The average discrimination power of concepts in different frequency intervals, _i.e._\(\alpha\in(0.0,0.2],(0.2,0.4],...,(0.8,1.0]\). The weighted average discrimination power \(\bar{\beta}\) over concepts of all frequencies is shown beside the curve.

Figure 7: The (left) transferability and (right) discrimination power of concepts decreased when we added more label noises.
patterns, it is difficult for a DNN to learn transferable concepts from noisy training samples.

_Experimental verification._ In this experiment, we injected noise into training samples to examine whether DNNs trained on datasets with noisy samples did not learn transferable concepts. Just like the experiment in "Case 1," we constructed such datasets by corrupting a clean dataset. To this end, we added Gaussian noises \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) to each input sample \(\mathbf{x}\) in the clean dataset by modifying it to \((1-\delta)\cdot\mathbf{x}+\delta\cdot\mathbf{\epsilon}\), where \(\delta\in[0,1]\) denotes the strength of noise injected into the sample \(\mathbf{x}\). Each dimension of the clean sample \(\mathbf{x}\) was normalized to unit variance over the dataset beforehand.

We constructed a series of datasets by injecting noises of different strength \(\delta\) into samples in the _wifi_ dataset3. We trained MLP-5's3 based on these datasets. Just like experiments in "Case 1," we examined the transferability and discrimination power of concepts. Fig. 8(left) shows that the transferability of concepts was usually low when the DNN was learned from noisy input data. Fig. 8(right) shows that the average discrimination power \(\bar{\beta}\) of concepts decreased along with the increasing strength \(\delta\) of injected noise. This verified that DNN usually did not learn transferable and discriminative concepts when input data were noisy.

Footnote 3: [https://github.com/](https://github.com/)
PointNet++ [Qi et al., 2017b]. Please see Appendix B.1 for the classification accuracy of the above DNNs.

## 4 Conclusion

In this paper, we have analyzed the interaction concepts encoded by a DNN. Specifically, we quantitatively examine the concept-emerging phenomenon of a DNN from four perspectives. Extensive empirical studies have verified that a well-trained DNN usually encodes sparse, transferable, and discriminative interaction concepts. Our experiments also prove the faithfulness of the interaction concepts extracted from DNNs. Besides, we also discussed three cases in which a DNN is unlikely to learn transferable concepts.
