Bayesian Neural Networks Avoid Encoding Complex and
  Perturbation-Sensitive Concepts
In this paper, we focus on mean-field variational Bayesian Neural Networks
(BNNs) and explore the representation capacity of such BNNs by investigating
which types of concepts are less likely to be encoded by the BNN. It has been
observed and studied that a relatively small set of interactive concepts
usually emerge in the knowledge representation of a sufficiently-trained neural
network, and such concepts can faithfully explain the network output. Based on
this, our study proves that compared to standard deep neural networks (DNNs),
it is less likely for BNNs to encode complex concepts. Experiments verify our
theoretical proofs. Note that the tendency to encode less complex concepts does
not necessarily imply weak representation power, considering that complex
concepts exhibit low generalization power and high adversarial vulnerability.
The code is available at https://github.com/sjtu-xai-lab/BNN-concepts.
Qihan Ren, Huiqi Deng, Yunuo Chen, Siyu Lou, Quanshi Zhang
# Bayesian Neural Networks Tend to Ignore Complex and Sensitive Concepts

###### Abstract

In this paper, we focus on mean-field variational Bayesian Neural Networks (BNNs) and explore the representation capacity of such BNNs by investigating which types of concepts are less likely to be encoded by the BNN. It has been observed and studied that a relatively small set of interactive concepts usually emerge in the knowledge representation of a sufficiently-trained neural network, and such concepts can faithfully explain the network output. Based on this, our study proves that compared to standard deep neural networks (DNNs), it is less likely for BNNs to encode complex concepts. Experiments verify our theoretical proofs. Note that the tendency to encode less complex concepts does not necessarily imply weak representation power, considering that complex concepts exhibit low generalization power and high adversarial vulnerability. _The code will be released when the paper is accepted._

## 1 Introduction

Unlike standard deep neural networks (DNNs), Bayesian neural networks (BNNs) represent network weights as probability distributions. Therefore, BNNs exhibit distinctive representation capacities from standard DNNs. Existing studies (Blundell et al., 2015; Gal and Smith, 2018; Kristiadi et al., 2020; Carbone et al., 2020; Wenzel et al., 2020; Krishnan et al., 2020; Zhang et al., 2022) usually analyzed BNNs in terms of generalization power, adversarial robustness, and optimization.

In contrast to the above studies, this paper proposes a new perspective to investigate the representation capacity of BNNs, _i.e._, we discover and theoretically prove that BNNs are less likely to encode sensitive and complex concepts than standard DNNs. In fact, such a property brings in specific advantages to feature representations of the BNN. To be precise, we limit our research to the scope of **mean-field variational BNNs**(Blundell et al., 2015), which is one of the most commonly used BNNs. Thus, in this paper, we just use the term \(BNN\) to refer to mean-field variational BNNs.

**Common phenomenon of concept emergence in various neural networks.** Although it is well-known that a neural network does not explicitly encode concepts like graphical models, recent studies (Ren et al., 2021, 2023; Deng et al., 2021) have discovered a common concept-emerging phenomenon that neural networks usually _implicitly_ encode a small number of interactive concepts for inference, which have been observed in different neural networks for various tasks. Specifically, each interactive concept represents an AND relationship among a set of input variables.

For example, we can use \(I(S=\{\text{eyes},\text{nose},\text{mouth}\})=U_{S\cdot\text{exist}(\text{eyes}) \cdot\text{exist}(\text{nose})\cdot\text{exist}(\text{mouse})}\) to illustrate the AND relationship for the face concept in image classification. If any image patch in the set \(S=\{\text{eyes},\text{nose},\text{mouth}\}\) is masked, then the face concept will be deactivated, and the numerical effect of this concept is removed (\(I(S)=0\)) and no longer influences the network output.

More importantly, interactive concepts can be considered as faithful inference patterns encoded by the neural network. It is because Ren et al. (2021) has proved that people can use a relatively small number of interactive concepts to well mimic the inference logic of the neural network on a certain input sample. That is, numerical effects of these concepts always well predict diverse network outputs, no matter how the input sample is masked.

**BNNs ignore complex and sensitive concepts.** Based on the interactive concepts, in this paper, we discover and theoretically prove that _compared to standard DNNs, it is more difficult for a neural network to encode complex interactive concepts, as long as it has weight uncertainty_. The complexity of an interactive concept \(S\) is defined as the number of variables in the set \(S\), _i.e._, \(\operatorname{complexity}(S)=|S|\). \(|S|\) is also termed the _order_ of the interactive concept.

We prove the above conclusion through three steps. First, it is difficult to theoretically analyze interactive concepts encoded by BNNs, because BNNs represent network weights as probability distributions. To this end, we find that we can usually use a _surrogate DNN model_, which is constructed by adding perturbations to both the input and low-layer features of a standard DNN, to approximate feature representations
of a BNN. In this way, we can directly analyze the surrogate DNN model with feature uncertainty, instead of investigating the BNN with weight uncertainty.

Second, we prove that in the surrogate DNN model, high-order interactive concepts are more sensitive to random perturbations than low-order interactive concepts.

Third, we prove that the sensitivity makes high-order interactive concepts difficult to be learned when features are perturbed. In this way, we can conclude that high-order interactive concepts are also less likely to be learned by the BNN when its weights are perturbed.

In addition, experiments showed that the strength of high-order (complex) interactive concepts encoded by BNNs was weaker than those encoded by standard DNNs, which verified the above theoretical conclusion.

**Note that our proof does NOT mean that a BNN has limited representation capacity.** Instead, we just demonstrate the distinctive tendency of avoiding encoding complex (high-order) interactive concepts, when weight uncertainty is introduced into the neural network. This does not mean that BNNs have weaker representation power than standard DNNs. If the task loss requires to encode complex concepts, then our research indicates that the BNN must reduce its weight uncertainty, to some extent.

**Practical values and advantages of avoiding encoding complex concepts.** Although we prove that BNNs tend to avoid encoding complex concepts, it is not necessarily a disadvantage of the BNN, compared to standard DNNs. On the contrary, it has been found that compared to simple (low-order) interactive concepts, complex (high-order) interactive concepts encoded by a neural network usually have poorer generalization ability (Lengerich et al., 2022) and are more vulnerable to adversarial attacks (Ren et al., 2021). Thus, encoding less complex concepts may be an advantage. Please see Appendix D for experiments that show the high adversarial vulnerability of high-order interactive concepts.

## 2 BNNs ignore complex and sensitive concepts

Unlike standard DNNs, a BNN represents each weight in the network as a probability distribution, instead of a scalar. In this paper, we limit the scope of our study to mean-field variational BNNs (Blundell et al., 2015), where all weights \(\mathbf{W}\) are formulated as a Gaussian distribution \(\mathcal{N}(\mathbf{W};\mathbf{\mu},\mathbf{\Sigma})\), and the covariance matrix \(\mathbf{\Sigma}\) is diagonal. Other types of BNNs (_e.g._, BNNs based on the Monte Carlo Dropout (Gal and Ghahramani, 2016)) are not discussed. The BNN learns parameters \(\mathbf{\theta}=(\mathbf{\mu},\mathbf{\Sigma})\), and we use \(q_{\mathbf{\theta}}(\mathbf{W})\) to represent the weight distribution. Let us consider a classification task with the training data \(\mathcal{D}=\{(\mathbf{x}^{(1)},y^{(1)}),\ldots,(\mathbf{x}^{(n)},y^{(n)})\}\). Training a BNN is to minimize the Kullback-Leibler (KL) divergence between the distribution \(q_{\mathbf{\theta}}(\mathbf{W})\) and the posterior distribution \(p(\mathbf{W}|\mathcal{D})\).

\[\mathbf{\theta}^{*} =\operatorname*{argmin}_{\mathbf{\theta}}\mathrm{KL}[q_{\mathbf{\theta}} (\mathbf{W})\|p(\mathbf{W}|\mathcal{D})] \tag{1}\] \[=\operatorname*{argmin}_{\mathbf{\theta}}-\mathbb{E}_{\mathbf{W}\sim q_{ \mathbf{\theta}}(\mathbf{W})}[\log p(\mathcal{D}|\mathbf{W})]+\mathrm{KL}[q_{\mathbf{\theta}} (\mathbf{W})\|p(\mathbf{W})],\]

where the first term is the classification loss, and the second term is the KL divergence between \(q_{\mathbf{\theta}}(\mathbf{W})\) and the prior distribution \(p(\mathbf{W})\), which is usually formulated as a Gaussian distribution \(\mathcal{N}(\mathbf{W};\mathbf{0},\mathbf{I})\). In addition, given a testing sample \(\mathbf{x}\), the inference of the BNN is conducted as follows. First, network weights are sampled from the weight distribution \(q_{\mathbf{\theta}}(\mathbf{W})\) to construct multiple neural networks. Then, each network is used to conduct inference on the sample \(\mathbf{x}\), and the final inference result \(p(y|\mathbf{x})\) is computed as the average classification probability of all the networks.

\[p(y|\mathbf{x})=\mathbb{E}_{\mathbf{W}\sim q_{\mathbf{\theta}}(\mathbf{W})}[p(y|\mathbf{x},\mathbf{W})] \tag{2}\]

### Preliminaries: emergence of sparse concepts

The learning of neural networks is usually regarded as a fitting problem between the ground-truth label and the

Figure 1: (a) Illustration of interactive concepts encoded by a neural network. Each interactive concept \(S\) corresponds to an AND relationship among a specific set \(S\) of input variables (image patches). \(C_{S}\) represents the activation state of the concept \(S\). The patches \(x_{1}\) and \(x_{6}\) are masked, so that concepts \(S_{1}\) and \(S_{3}\) are deactivated, _i.e._, \(C_{S_{1}}=0\) and \(C_{S_{3}}=0\). (b) Experiments demonstrate the common concept-emerging phenomenon. Neural networks with various architectures all encode sparse interactive concepts. In other words, most interactive concepts have near-zero effects, _i.e._, \(I(S)\approx 0\), and can be considered as noises; only a relatively small number of interactive concepts have significant effects. For better visualization, the interactive concepts are sorted by strengths in descending order.
model prediction, which does not require explicit learning of specific concepts. However, recent studies (Ren et al., 2021, 2023; Deng et al., 2021) have empirically discovered that in many tasks, sparse AND relationships between input variables were usually implicitly encoded by a neural network, when the neural network was sufficiently trained. As shown in Figure 1(a), these AND relationships can be viewed as specific types of _interactive concepts_, which will be introduced in the **interactive concepts** paragraph.

Although counter-intuitive, this concept-emerging phenomenon does exist in various neural networks. Furthermore, such interactive concepts have been used to prove the representation bottleneck of the neural network (Deng et al., 2021) and obtain optimal masking states for attribution methods (Ren et al., 2023). **We also verify the trustworthiness of using interactive concepts to explain neural networks in experiments (see the end of this section).**

**Interactive concepts.**Ren et al. (2021) proposed the interaction effect \(I(S)\) to study the emergence of concepts. Let us consider a pre-trained neural network \(v\) and an input sample \(\mathbf{x}=[x_{1},\ldots,x_{n}]\) with \(n\) input variables indexed by \(N=\{1,\ldots,n\}\). Let \(\Omega\) denote a set of interactive concepts extracted from the network. Each interactive concept \(S\in\Omega\) corresponds to the collaboration (AND relationship) between input variables in a specific set \(S\subseteq N\), thus \(\Omega\subseteq 2^{N}=\{S|S\subseteq N\}\). For instance, as Figure 1(a) shows, a concept \(S=\{x_{1},x_{2},x_{3}\}\) is formed due to the co-occurrence of the three image patches. The concept will be activated and make a certain interaction effect \(I(S)\) on the network output, only if the patches \(x_{1},x_{2},x_{3}\) are all present. In contrast, the absence (masking) of any patch among \(x_{1}\), \(x_{2}\), and \(x_{3}\) will deactivate the concept and remove the interaction effect, _i.e._, \(I(S|\mathbf{x}^{\text{mask}})=0\).

Specifically, the interaction effect \(I(S|\mathbf{x})\) on the sample \(\mathbf{x}\) is computed by the Harsanyi dividend1(Harsanyi, 1963).

Footnote 1: Please see Appendix E for properties and trustworthiness of the Harsanyi dividend.

\[I(S|\mathbf{x})=\sum\nolimits_{T\subseteq S}(-1)^{|S|-|T|}\cdot v(\mathbf{x}_{T}). \tag{3}\]

If \(I(S|\mathbf{x})\) has a significant value, then the neural network is considered to encode an interactive concept \(S\); otherwise, if \(I(S|\mathbf{x})\approx 0\), the concept \(S\) does not exist. Here, \(\mathbf{x}_{T}\) denotes the masked input sample, where variables in \(N\setminus T\) are masked and variables in \(T\) are kept unchanged. Besides, \(v(\mathbf{x}_{T})\in\mathbb{R}\) can be computed as a scalar output of the neural network on the masked sample \(\mathbf{x}_{T}\) (_e.g._, the confidence score of classifying the input sample \(\mathbf{x}_{T}\) to the ground-truth category \(v(\mathbf{x}_{T})=\text{log}\frac{p(y=y_{\text{data}}|\mathbf{x}_{T})}{1-p(y=y_{ \text{data}}|\mathbf{x}_{T})}\)).

**Faithfulness of interactive concepts.** Given an input sample \(\mathbf{x}\) with \(n\) variables, we have \(2^{n}\) different ways to mask the sample \(\mathbf{x}\) and obtain the masked sample \(\mathbf{x}_{T}\)_w.t.t._ all subsets \(T\subseteq N\). To this end, Ren et al. (2021) proved that

\[\exists\ \Omega\subseteq 2^{N},\ s.t.\ \forall\ T\subseteq N,v(\mathbf{x}_{T})= \sum\nolimits_{S\in\Omega,S\subseteq T}I(S|\mathbf{x}), \tag{4}\]

where \(2^{N}=\{S|S\subseteq N\}\). The equation indicates that interactive concepts in \(\Omega\) can well mimic network outputs on all the \(2^{n}\) masked samples. Thus, we can consider that all interactive concepts in the set \(\Omega\) as faithful inference patterns encoded by the neural network.

**Sparsity of interactive concepts.** More crucially, extensive experiments (Ren et al., 2021; Deng et al., 2021; Ren et al., 2023) discovered that interactive concepts emerging in a neural network are usually very sparse. Figure 1(b) shows that most interactive concepts have near-zero interaction effects (\(|I(S|\mathbf{x})|\approx 0\)), thus having negligible influence on the network output. Only a few salient interactive concepts have significant effects \(I(S|\mathbf{x})\) on the network output. In this way, the network output can be mimicked by only a few salient interactive concepts in \(\Omega_{\text{salient}}\).

\[\forall\ T\subseteq N,\ \ v(\mathbf{x}_{T})=\sum\nolimits_{S\in\Omega_{\text{salient}}}I(S|\mathbf{x})+\epsilon \tag{5}\]

The above equation decomposes the output \(v(\mathbf{x}_{T})\) into two parts: (1) effects of all salient interactive concepts in \(\Omega_{\text{salient}}\), and (2) a small residual term \(\epsilon\) containing negligible effects of all non-salient interactive concepts.

**Empirically verifying the sparsity of concepts.** Based on Eq. (5), in the following analysis, _only salient interactive concepts in \(\Omega_{\text{salient}}\) are regarded as valid concepts encoded by a neural network._ We empirically verify the emergence of sparse concepts in various neural networks, including multi-layer perceptrons (MLPs), residual multi-layer perceptrons (ResMLPs) (Touvron et al., 2022), long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997), and convolutional neural networks (CNNs), and on different datasets, including tabular data (Census dataset and TV news dataset (Dua and Graff, 2017)), language data (CoLA (Warstadt et al., 2019) and SST-2 (Socher et al., 2013)), and image data (MNIST (LeCun et al., 1998)). (1) Figure 1(b) verifies that concepts encoded by various neural networks are all sparse. (2) Appendix G verifies that the training process boosts the sparsity of concepts.

**Complexity of a neural network representing a concept.** In many previous studies (Deng et al., 2021; Wang et al., 2021; Zhang et al., 2020), the complexity of an interactive concept \(S\) was measured by the number of variables in the set \(S\) (also termed the _order_ of the interactive concept), _i.e._, \(\operatorname{complexity}(S)=\operatorname{order}(S)=|S|\). Then, a low-order concept represents a simple collaboration among a few input variables, while a high-order concept represents a complex collaboration among many input variables.
### Approximating weight uncertainty by adding input perturbations

In this paper, we aim to prove that compared to standard DNNs, it is more difficult to encode high-order (complex) interactive concepts as long as the network has weight uncertainty. Note that previous studies [10, 22] found that a DNN encoding less complex concepts was **NOT** necessarily equivalent to a weak representation capacity. Instead, it usually boosts the generalization power and adversarial robustness. Please see Appendix D for experiments on the high adversarial vulnerability of high-order concepts. In addition, as discussed in the last two paragraphs of the introduction, the BNN can still encode complex concepts when it learns small variances.

Unlike standard DNNs, a BNN formulates each weight as a probability distribution, which boosts the difficulty of theoretically analyzing interactive concepts encoded in a BNN. Therefore, in this subsection, we first discover that introducing uncertainty to weights in the BNN can be approximated by adding perturbations to input variables and low-layer features in experiments. In other words, we add random perturbations to both input variables and low-layer features of a standard DNN, and we demonstrate that such a perturbed DNN performs as a _surrogate DNN model_, which well approximates feature representations of a BNN.

Let us consider a feed-forward BNN, which has \(L\) cascaded linear layers and ReLU layers. Given an input sample \(\mathbf{x}\in\mathbb{R}^{P_{0}}\) (\(D_{0}=n\)), the feature of the \(l\)-th layer \(\mathbf{h}^{(l)}\in\mathbb{R}^{D_{l}}\) (\(1\leq l\leq L\)) is computed as follows.

\[\mathbf{h}^{(l)}=\mathbf{W}^{(l)}(\cdots\mathbf{\Phi}^{(1)}(\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1 )})\cdots)+\mathbf{b}^{(l)}, \tag{6}\]

where \(\mathbf{W}^{(l)}\in\mathbb{R}^{D_{l}\times D_{l-1}}\) and \(\mathbf{b}^{(l)}\in\mathbb{R}^{D_{l}}\) denote the weight matrix and bias of the \(l\)-th linear layer, respectively. In the BNN, \(W^{(l)}_{ij}\sim\mathcal{N}(\overline{W}^{(l)}_{ij},(\sigma^{(l)}_{ij})^{2})\) is independently sampled from Gaussian distributions. We use \(\mathbf{\mu}_{\mathbf{W}^{(l)}}=[\overline{W}^{(l)}_{ij}]\in\mathbb{R}^{D_{l}\times D _{l-1}}\) to denote the mean of the weight matrix. Besides, \(\mathbf{b}^{(l)}\sim\mathcal{N}(\mathbf{\mu}_{\mathbf{b}^{(l)}},\mathbf{\Sigma}_{\mathbf{b}^{(l)}})\), where \(\mathbf{\Sigma}_{\mathbf{b}^{(l)}}\) is a diagonal matrix. The diagonal matrix \(\mathbf{\Phi}^{(l)}=\mathrm{diag}(\phi^{(l)}_{1},\cdots,\phi^{(l)}_{D_{l}})\in \{0,1\}^{D_{l}\times D_{l}}\) denotes binary gating states of the \(l\)-th ReLU layer.

Then, we construct the surrogate DNN model with the same architecture as the BNN, to approximate the BNN's feature distribution. Parameters of this surrogate DNN model \(\mathbf{\psi}\) are set as the mean of the weight distribution and the mean of the bias distribution in the BNN, _i.e_., \(\mathbf{\psi}=\{\mathbf{\mu}_{\mathbf{W}^{(l)}},\mathbf{\mu}_{\mathbf{b}^{(l)}}\}_{L=1}^{L}\). Given an input sample \(\mathbf{x}\), we add perturbations \(\Delta\mathbf{x}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma}_{\Delta\mathbf{x}})\) to input variables and perturbations \(\Delta\mathbf{h}^{(l^{\prime})}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma}_{\Delta\mathbf{h}^ {(l^{\prime})}})\) to features between the first layer and the \((l-1)\)-th layer in the surrogate DNN model (\(1\leq l^{\prime}\leq l-1\)). In this way, we can obtain the distribution of the \(l\)-th layer feature \(\tilde{h}^{(l)}\) in the surrogate DNN model, denoted as \(p_{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\
BNN, we constructed a corresponding surrogate DNN model. Please see Appendix I for implementation details.

Figure 2 shows that the feature distribution of the surrogate DNN model well matched the feature distribution of the BNN. Furthermore, we used the KL divergence \(\mathrm{KL}(p_{\text{BNN}}(\mathbf{h}^{(l)})\|p_{\text{DNN}}(\mathbf{\tilde{h}}^{(l)}|\mathbf{ \Delta}))\) in Eq. (7) to measure the approximation error. To compare with \(\mathrm{KL}(p_{\text{BNN}}(\mathbf{h}^{(l)})\|p_{\text{DNN}}(\mathbf{\tilde{h}}^{(l)}| \mathbf{\Delta}))\), we further constructed a simple baseline distribution of the features \(p_{\text{base}}(\mathbf{h}^{(l)})=\mathcal{N}(\hat{\mathbf{\mu}}\hat{1},\hat{\sigma}^{ 2}\mathbf{I})\), where \(\hat{\mu}\) and \(\hat{\sigma}^{2}\) denote the mean and the variance over all feature dimensions of the BNN, respectively. We computed \(\mathrm{KL}(p_{\text{BNN}}(\mathbf{h}^{(l)})\|p_{\text{base}}(\mathbf{h}^{(l)}))\) for comparison. Table 1 shows that the approximation error of the surrogate DNN model was significantly smaller than the approximation error of the baseline distribution.

Experimental results showed that the weight uncertainty in a BNN could be well approximated by adding random perturbations to both input variables and low-layer features.

### High-order concepts are sensitive

In this subsection, we theoretically prove that high-order interactive concepts are more sensitive to perturbations than low-order interactive concepts. In the next subsection, we will prove that sensitive interactive concepts are difficult to be learned by a neural network.

Note that according to Section 2.2, introducing the weight uncertainty in a BNN can be approximated by adding random perturbations to both input variables and features of different layers. However, simultaneously adding perturbations to features of multiple layers significantly boosts the difficulty of analysis. Fortunately, adding perturbations to output features of the \(l\)-th layer can be considered as perturbing input variables of the \((l+1)\)-th layer. Hence, in this subsection, we just analyze interactive concepts in a simple case where we perturb input variables in a certain layer, instead of analyzing the complex case of simultaneously perturbing features of different layers.

To prove that high-order interactive concepts are more sensitive to input perturbations than low-order interactive concepts, let us first derive the analytical form of the interaction effect \(I(S)\) of an interactive concept.

**Lemma 2.1** (Proof in Appendix F.1).: _Given a neural network \(v\) and an arbitrary input sample \(\mathbf{x}^{\prime}\in\mathbb{R}^{n}\), the network output can be decomposed using the Taylor expansion \(v(\mathbf{x}^{\prime})=\sum_{S\in N}\sum_{\mathbf{x}\in Q_{S}}U_{S,\mathbf{\pi}}\cdot J(S, \mathbf{\pi}|\mathbf{x}^{\prime})\). In this way, according to Eq. (3), the interaction effect \(I(S|\mathbf{x}^{\prime})\) on the sample \(\mathbf{x}^{\prime}\) can be reformulated as_

\[I(S|\mathbf{x}^{\prime})=\sum\nolimits_{\mathbf{\pi}\in Q_{S}}U_{S,\mathbf{\pi}}\cdot J(S, \mathbf{\pi}|\mathbf{x}^{\prime}), \tag{8}\]

_where \(J(S,\mathbf{\pi}|\mathbf{x}^{\prime})=\prod_{i\in S}\left(\mathrm{sign}(x^{\prime}_{i }-r_{i})\cdot\frac{x^{\prime}_{i}-r_{i}}{\tau}\right)^{x_{i}}\) denotes an expansion term of the degree \(\mathbf{\pi}\), \(\mathbf{\pi}\in Q_{S}=\{[\pi_{1},\ldots,\pi_{n}]|\forall i\in S,\pi_{i}\in\mathbb{ N}^{+};\forall i\not\in S,\pi_{i}=0\}\). \(U_{S,\mathbf{\pi}}\)= \(\frac{\tau^{m}}{\prod_{i=1}^{n}\pi_{i}!}\frac{\partial^{m}\pi(\mathbf{x}_{i})}{ \partial x^{n_{1}}_{1}\cdots\partial x^{m_{n}^{n}}_{m}}\cdot\prod_{i\in S}[ \mathrm{sign}(x^{\prime}_{i}-r_{i})]^{\pi_{i}}\), \(m=\sum_{i=1}^{n}\pi_{i}\)._

Lemma 2.1 provides a new perspective to analyze the sensitivity of the interaction effect \(I(S)\). In particular, just like in Ren et al. (2021) and Ren et al. (2023), we mask the input variable \(x_{i}\) by setting it to its reference value \(x_{i}\gets r_{i}\). The reference value \(r_{i}\) is designed as follows. Let \(\mathbb{E}_{\mathbf{x}}[x_{i}]\) denote the average value of the input variable \(x_{i}\) over all input samples, which is usually regarded as a no-information state of this input variable (Ancona et al., 2019). In this paper, we remove the information from the input variable \(x_{i}\) by pushing \(x_{i}\) by a large enough distance \(\tau\) towards its mean value. In other words, if \(x_{i}>\mathbb{E}_{\mathbf{x}}[x_{i}]\), we set the reference value \(r_{i}=x_{i}-\tau^{2}\); otherwise, \(r_{i}=x_{i}+\tau^{1}\). Here, \(\tau\in\mathbb{R}\) is a pre-defined constant. In this way, compared to setting \(r_{i}=\mathbb{E}_{\mathbf{x}}[x_{i}]\), the above setting ensures comparable perturbation magnitudes over different input dimensions.

Furthermore, in order to simplify the proof, when we add a small Gaussian perturbation \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})\) to the sample \(\mathbf{x}\), we ignore the extremely low possibility of large perturbations \(|\epsilon_{i}|\geq\tau\) because the variance \(\sigma^{2}\) is small.

Let us start with a simple case in Lemma 2.1. Since people usually adopt low-order Taylor expansion for approximation in real implementations, we first approximate the interaction effect \(I(S|\mathbf{x}^{\prime})\) using the expansion term of the lowest degree, and analyze the influence of input perturbations on \(I(S|\mathbf{x}^{\prime})\).

**Theorem 2.2** (Proof in Appendix F.2).: _Let \(\mathbf{\hat{\pi}}\) denote the lowest degree of the expansion terms of the interaction effect \(I(S|\mathbf{x}^{\prime})\), i.e., \(\forall i\in S,\hat{\pi}_{i}=1;\forall i\not\in S,\hat{\pi}_{i}=0\). Let us consider the interaction effect \(I(S|\mathbf{x}^{\prime})\) only containing the expansion term of the lowest degree, i.e., \(I(S|\mathbf{x}^{\prime})=U_{S,\mathbf{\hat{\pi}}}\cdot J(S,\mathbf{\hat{\pi}}|\mathbf{x}^{ \prime})\). In this way, the mean and variance of the interaction effect \(I(S|\mathbf{x}^{\prime}=\mathbf{x}+\mathbf{\epsilon})\) over different perturbations \(\mathbf{\epsilon}\) are given as_

\[\begin{split}\mathbb{E}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})] &=U_{S,\mathbf{\hat{\pi}}},\\ \mathrm{Var}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]& =U_{S,\mathbf{\hat{\pi}}}^{2}((1+(\sigma/\tau)^{2})^{|S|}-1).\end{split} \tag{9}\]

Theorem 2.2 proves that _the variance \(\mathrm{Var}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]\) increases along with the order \(|S|\) of the interactive concept in an exponential manner_. It indicates that high-order interactive concepts are much more sensitive to input perturbations than low-order concepts. Furthermore, as mentioned in Section 2.2, since we can add perturbations to a surrogate DNN model to well mimic feature representations of a BNN, **we can consider that high-order interactive concepts encoded by the BNN are much more sensitive to weight uncertainty in the BNN than low-order concepts.**
**Theorem 2.3** (Proof in Appendix F.3).: _Let \(\mathbf{\pi}\in Q_{S}=\{[\pi_{1},\ldots,\pi_{n}]|\forall i\in S,\pi_{i}\in\mathbb{N}^{+} ;\forall i\not\in S,\pi_{i}=0\}\) denote an arbitrary degree. Then, the mean and the variance of \(J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})\) over perturbations \(\mathbf{\epsilon}\) are_

\[\mathbb{E}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})] =\mathbb{E}_{\mathbf{\epsilon}}[\prod\nolimits_{i\in S}(1+\frac{ \epsilon_{i}}{\tau})^{\pi_{i}}], \tag{10}\] \[\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})] =\mathrm{Var}_{\mathbf{\epsilon}}[\prod\nolimits_{i\in S}(1+\frac{ \epsilon_{i}}{\tau})^{\pi_{i}}]\]

Theorem 2.3 extends Theorem 2.2 to a **general case**, where we use a higher-order Taylor expansion to represent \(I(S|\mathbf{x}^{\prime})\).

**Theorem 2.4** (Proof in Appendix F.4).: _Let \(S\) and \(S^{\prime}\) be two interactive concepts, such that \(S\subseteq S^{\prime}\). Let us consider expansion terms \(J(S,\mathbf{\pi})\) and \(J(S^{\prime},\mathbf{\pi}^{\prime})\), where the term \(J(S^{\prime},\mathbf{\pi}^{\prime})\) is extended from the term \(J(S,\mathbf{\pi})\) with \(\mathbf{\pi}\prec\mathbf{\pi}^{\prime}\). I.e., (1) \(\forall i\in S^{\prime},\pi^{\prime}_{i}\in\mathbb{N}^{+}\); otherwise, \(\pi^{\prime}_{i}=0\). (2) Given \(\mathbf{\pi}^{\prime}\), \(\forall j\in S,\pi_{j}=\pi^{\prime}_{j}\); otherwise, \(\pi_{j}=0\). Then, we have_

\[\frac{\mathrm{Var}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{\prime }|\mathbf{x}+\mathbf{\epsilon})]}{\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\bm {\epsilon})]}>\prod\nolimits_{i\in S^{\prime}\setminus S}\mathbb{E}_{\epsilon_ {i}}^{2}[(1+\frac{\epsilon_{i}}{\tau})^{\pi^{\prime}_{i}}], \tag{11}\] \[\frac{\mathbb{E}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{\prime}| \mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{ \prime}|\mathbf{x}+\mathbf{\epsilon})]}{\mathbb{E}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x }+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{ \epsilon})]}\] \[<\frac{1}{\prod\nolimits_{i\in S^{\prime}\setminus S}\mathbb{E}_{ \epsilon_{i}}[(1+\frac{\epsilon_{i}}{\tau})^{\pi^{\prime}_{i}}]},\]

_and we can also obtain \(\mathbb{E}_{\epsilon_{i}}[(1+\frac{\epsilon_{i}}{\tau})^{\pi^{\prime}_{i}}]\geq 1\)._

Theorem 2.4 indicates that for an arbitrary degree \(\mathbf{\pi}\) of the interactive concept \(\mathbf{S}\), \(\mathrm{Var}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{\prime}|\mathbf{x}+\mathbf{ \epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})]\) increases in an exponential manner along with \(|S^{\prime}\setminus S|=|S^{\prime}|-|S|\). Therefore, we can roughly consider that \(\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})]\) increases exponentially _w.r.t._ the order \(|S|\). Furthermore, according to Lemma 2.1, \(I(S|\mathbf{x}+\mathbf{\epsilon})\) can be re-written as the weighted sum of \(J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})\). Since coefficients \(U_{S,\mathbf{\pi}}\)_w.r.t._ different \(S\) and \(\mathbf{\pi}\) are usually chaotic, we can roughly consider that the sensitivity of \(I(S|\mathbf{x}+\mathbf{\epsilon})\) also grows exponentially along with the order \(|S|\) of the interactive concept \(S\). In addition, Theorem 2.4 also proves the approximately exponential decrease of \(\frac{\mathbb{E}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{\prime}|\mathbf{x}+\mathbf{ \epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[J(S^{\prime},\mathbf{\pi}^{\prime}|\mathbf{x}+ \mathbf{\epsilon})]}{\mathbb{E}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{ \epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[J(S,\mathbf{\pi}|\mathbf{x}+\mathbf{\epsilon})]}\) along with \(|S^{\prime}|-|S|\). Similarly, we can obtain that the relative stability \(\mathbb{E}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{ \epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]\) decreases along with the order \(|S|\).

**Conclusions.** Both Theorem 2.2 and Theorem 2.4 tell us that high-order interactive concepts are much more sensitive to input perturbations. Furthermore, combined with the conclusion in Section 2.2, **we can conclude that high-order interactive concepts encoded by the BNN are much more sensitive to the weight uncertainty in the BNN than low-order concepts.

**Experimental verification.** We conducted experiments to verify the above conclusions. To verify the sensitivity to input perturbations, we added a random perturbation \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\sigma^{2}I)\) to a given input sample \(\mathbf{x}\), where \(\sigma^{2}=0.05^{2}\). Then, we used the following two metrics, \(V^{(s)}_{\text{noise}}=\mathbb{E}_{\mathbf{\epsilon}}[\mathbb{E}_{|S|=s}[\mathrm{ Var}_{\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\sigma^{2}I)}[I(S|\mathbf{x}+\mathbf{ \epsilon})]]]\) and \(K^{(s)}_{\text{noise}}=\mathbb{E}_{\mathbf{\epsilon}}[\mathbb{E}_{|S|=s}[\frac{ \mathbb{E}_{\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\sigma^{2}I)}[I(S|\mathbf{x}+\mathbf{ \epsilon})]]}{\mathrm{Var}_{\mathbf{\epsilon}}\sim\mathcal{N}(\mathbf{0},\sigma^{2}I) (I(S|\mathbf{x}+\mathbf{\epsilon})]}]\), to measure the average variance and the average relative stability of the \(s\)-order interactive concepts _w.r.t._ the input perturbation \(\mathbf{\epsilon}\). Then, a large \(V^{(s)}_{\text{noise}}\) or a small \(K^{(s)}_{\text{noise}}\) indicated that the \(s\)-order interactive concepts were sensitive to input perturbations.

Similarly, to verify the sensitivity to the weight uncertainty, we sampled different weights \(\mathbf{W}\) from the weight distribution \(q_{\mathbf{\theta}}(\mathbf{W})\) of the BNN. Then, we used \(V^{(s)}_{\text{BNN}}=\mathbb{E}_{\mathbf{\pi}}[\mathbb{E}_{|S|=s}[\mathrm{Var}_{ \mathbf{W}\sim q_{\mathbf{\theta}}(\mathbf{W})}[I(S|\mathbf{x},\mathbf{W})]]]\) and \(K^{(s)}_{\text{BNN}}=\mathbb{E}_{\mathbf{\pi}}[\mathbb{E}_{|S|=s}[\frac{\mathbb{E}_{ \mathbf{W}\sim q_{\mathbf{\theta}}(\mathbf{W})}[I(S|\mathbf{x},\mathbf{W})]]}{\mathrm{Var}_{\mathbf{ W}\sim q_{\mathbf{\theta}}(\mathbf{W})}[I(S|\mathbf{x},\mathbf{W})]}]\) to measure the average variance and the average relative stability of the \(s\)-order interactive concepts _w.r.t._ the weight uncertainty in the BNN. Therefore, a large value of \(V^{(s)}_{\text{BNN}}\) or a small value of \(K^{(s)}_{\text{BNN}}\) indicated that the \(s\)-order interactive concepts were sensitive to the weight uncertainty. We followed experimental settings in the _experiments_ paragraph in Section 2.2 to train BNNs. Specifically, we trained BNNs with the MLP architecture on the MNIST dataset, the TV news dataset, and the Census dataset. We trained BNNs with the LeNet architecture on the CIFAR-10 dataset. Appendix I introduces how to efficiently compute \(I(S|\mathbf{x})\) on images.

Figure 3 shows that the average variance \(V^{(s)}_{\text{noise}}\) and \(V^{(s)}_{\text{BNN}}\) increased exponentially along with the order \(s\), while the relative stability \(K^{(s)}_{\text{noise}}\) and \(K^{(s)}_{\text{BNN}}\) both decreased along with

Figure 3: (a) The exponential increase of the average variance \(V^{(s)}_{\text{noise}}\) and (b) the roughly exponential decrease of the average relative stability \(K^{(s)}_{\text{noise}}\) along with the order \(s\), under perturbations from a distribution \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},0.05^{2}\cdot\mathbf{I})\). (c) The exponential increase of the average variance \(V^{(s)}_{\text{BNN}}\) and (d) the roughly exponential decrease of the average relative stability \(K^{(s)}_{\text{BNN}}\) along with the order \(s\), under weight uncertainty in the BNN.
the order. This demonstrated that high-order interactive concepts were much more sensitive to input perturbations and the weight uncertainty in the BNN, thereby verifying Theorem 2.2 and Theorem 2.4.

### Sensitive concepts are difficult to learn

In this subsection, we prove that high-order interactive concepts, which are sensitive to input perturbations and weight uncertainty, are difficult to be learned by a BNN in a regression task. Specifically, we measure the learning effects of interactive concepts (denoted by \(U_{S}\)), and Theorems 2.6 and 2.7 prove the small learning effects of sensitive concepts.

To facilitate the analysis, we first simplify the conceptual learning as a linear problem. Specifically, we first rewrite the interaction effect of an interactive concept \(S\). Given an input sample \(\mathbf{x}\), according to Eq. (8), the interaction effect of the concept \(S\) on the sample \(\mathbf{x}^{\prime}\) (obtained by applying some transformations on \(\mathbf{x}\)), \(I(S|\mathbf{x}^{\prime})\), can be rewritten as

\[I(S|\mathbf{x}^{\prime})=U_{S}\cdot C_{S}(\mathbf{x}^{\prime}), \tag{12}\]

where the constant \(U_{S}=I(S|\mathbf{x})\) denotes the interaction effect of the concept \(S\), and the function for the activation state is given as \(C_{S}(\mathbf{x}^{\prime})=\sum_{\mathbf{\pi}\in Q_{S}}U_{S,\mathbf{\pi}}J(S,\mathbf{\pi}|\mathbf{ x}^{\prime})/U_{S}\).

**Theorem 2.5** (Proof in Appendix F.5).: _Given an arbitrarily masked sample \(\mathbf{x}_{T}(\forall T\subseteq N)\), the function \(C_{S}(\mathbf{x}_{T})\) defined above can well fit the binary activation state of the concept \(S\) in the sample \(\mathbf{x}_{T}\)._

\[\forall\,T\subseteq N,\;C(S|\mathbf{x}_{T})=\prod_{i\in S}A_{i}(\mathbf{x}_{T})= \mathbb{1}(S\subseteq T), \tag{13}\]

_where \(A_{i}(\mathbf{x}_{T})\in\{0,1\}\) denotes whether the variable \(x_{i}\) is present or being masked in the sample \(\mathbf{x}_{T}\)._

Theorem 2.5 shows that the function \(C_{S}(\mathbf{x}_{T})\) actually represents the AND relationship of the concept \(S\) under an arbitrarily masking condition \(\forall\;T\subseteq N\). Only when all variables in \(S\) are all present under the masking condition \(T\), the concept \(S\) is activated \(C_{S}(\mathbf{x}_{T})=1\). If any of variables in \(S\) is masked, then the concept \(S\) will not be activated \(C_{S}(\mathbf{x}_{T})=0\), yielding zero interaction effect \(I(S|\mathbf{x}_{T})=0\).

Thus, we can extend Eq. (4) to a continuous version that explains the output as a **linear regression problem**.

\[v(\mathbf{x}^{\prime})=\sum\nolimits_{S\in\Omega}U_{S}\cdot C_{S}(\mathbf{x}^{\prime}), \tag{14}\]

where the activation state \(C_{S}(\mathbf{x}^{\prime})\) can be considered as an input dimension of the linear function, which reflects whether the input sample \(\mathbf{x}^{\prime}\) contains the concept \(S\).

Therefore, the absolute value of the coefficient \(U_{S}\) can be considered as _the strength of the neural network in learning the interactive concept \(S\)_. According to Section 2.1 and Ren et al. (2021), most interactive concepts have negligible coefficients \(|U_{S}|\approx 0\), so we can consider that the neural network only encodes a few interactive concepts \(S\) with large absolute values \(|U_{S}|\).

Let us facilitate the poof on a regression task. Based on the conclusion in Section 2.2, we can roughly consider that training a BNN on normal samples is equivalent to training a surrogate DNN model on perturbed input samples \(\mathbf{x}^{\prime}=\mathbf{x}+\mathbf{\epsilon}\). Then, according to Eq. (14), the learning of the BNN on a certain input sample can be roughly represented as \(\min_{\{U_{S}|S\in\Omega\}}L(\{U_{S}\})\), and the loss is given by

\[\begin{split} L(\{U_{S}\})&=\mathbb{E}_{\mathbf{\epsilon }}\left[(y^{*}-v(\mathbf{x}^{\prime}))^{2}\right]\\ &=\mathbb{E}_{\mathbf{\epsilon}}[(y^{*}-\sum\nolimits_{S\in\Omega} U_{S}\cdot C_{S}(\mathbf{x}+\mathbf{\epsilon}))^{2}]\end{split} \tag{15}\]

where \(\mathbf{x}\) and \(y^{*}\) denote the input sample and the ground-truth output, respectively, and \(\mathbf{x}^{\prime}=\mathbf{x}+\mathbf{\epsilon}\).

**Theorem 2.6** (Proof in Appendix F.6).: _Given two random interactive concepts \(S\) and \(S^{\prime}\), we can roughly assume that \(C_{S}(\mathbf{x}+\mathbf{\epsilon})\) is independent of \(C_{S^{\prime}}(\mathbf{x}+\mathbf{\epsilon})\), because the two concepts \(S\) and \(S^{\prime}\) usually have little overlap in most cases. Let \(\mathbb{E}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]\) and \(\mathrm{Var}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]\) denote the mean and the variance of \(C_{S}(\mathbf{x}+\mathbf{\epsilon})\) w.r.t. \(\mathbf{\epsilon}\), respectively. Then, the solution to Eq. (15) satisfies the following property:_

\[\forall\;S\in\Omega,\quad|U_{S}^{*}|\propto|\mathbb{E}_{\mathbf{\epsilon}}[C_{S}( \mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]| \tag{16}\]

Theorem 2.6 proves that the learning effect of an interactive concept \(S\), measured by \(|U_{S}^{*}|\), is proportional to the relative stability of the activation state of the interactive concept \(|\mathbb{E}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{ \epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]|\) w.r.t. perturbations \(\mathbf{\epsilon}\). This indicates that sensitive interactive concepts are more difficult to learn. The experimental verification of this theorem is shown in Appendix H.

**Theorem 2.7** (Proof in Appendix F.7).: _Let \(A^{\text{min}}=\min_{S}|U_{S}|\) and \(A^{\text{max}}=\max_{S}|U_{S}|\) denote the lower bound and the upper bound of \(|U_{S}|\) over all interactive concepts \(S\). Then, for any \(S\subseteq N\), we have_

\[\begin{split} A^{\text{min}}\cdot\frac{|\mathbb{E}_{\mathbf{\epsilon }}[I(S|\mathbf{x}+\mathbf{\epsilon})]|}{\mathrm{Var}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{ \epsilon})]}&\leq\frac{|\mathbb{E}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+ \mathbf{\epsilon})]|}{\mathrm{Var}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]}\\ &\leq A^{\text{max}}\cdot\frac{|\mathbb{E}_{\mathbf{\epsilon}}[I(S| \mathbf{x}+\mathbf{\epsilon})]|}{\mathrm{Var}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})] }\end{split} \tag{17}\]

Theorem 2.7 proves that high-order (complex) interactive concepts have low relative stability _w.r.t._ perturbations \(\mathbf{\epsilon}\). In fact, both Theorem 2.4 and Figure 3 have told us that \(|\mathbb{E}_{\mathbf{\epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{ \epsilon}}[I(S|\mathbf{x}+\mathbf{\epsilon})]|\) significantly decreases along with the order \(s=|S|\) of the interactive concept \(S\). Therefore, both the lower bound and the upper bound of \(|\mathbb{E}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]/\mathrm{Var}_{\mathbf{ \epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]|\) in Eq. (17) decrease along with the order \(s\) significantly. In this way, we can approximately consider that the strength of encoding a concept \(|U_{S}^{*}|\propto|\mathbb{E}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]/ \mathrm{Var}_{\mathbf{\epsilon}}[C_{S}(\mathbf{x}+\mathbf{\epsilon})]|\) also decreases along
with the order of interactive concepts. In other words, we prove that high-order interactive concepts are more difficult to be learned under perturbations \(\epsilon\). Combining the conclusion in Section 2.2, we also prove that high-order interactive concepts are more difficult to be learned by the BNN.

## 3 Experiments

In this section, we experimentally verified that compared to standard DNNs, BNNs were less likely to encode high-order (complex) interactive concepts. Specifically, we constructed three pairs of baseline networks for comparison.

(1) Given a trained BNN \(\mathbf{\theta}^{*}\), we constructed a standard DNN by setting its weights to the mean value of the weight distribution of the BNN. The standard DNN was denoted by \(\mathbf{\psi}_{\mathbf{\theta}^{*}}\). Then, we compared the strength of all high-order interactive concepts between the BNN \(\mathbf{\theta}^{*}\) and the standard DNN \(\mathbf{\psi}_{\mathbf{\theta}^{*}}\) without weight/feature uncertainty.

(2) Similarly, given a trained standard DNN \(\mathbf{\psi}^{*}\), we constructed a BNN \(\mathbf{\theta}_{\psi^{*}}\) by setting the mean value of its weight distribution to the weights of the standard DNN. We set all weight dimensions in the \(l\)-th layer of the BNN to share the same variance \(\sigma_{l}^{2}\), where \(\sigma_{l}^{2}\) was computed as the average of variances of all weight dimensions in the \(l\)-th layer of the previous BNN \(\mathbf{\theta}^{*}\). Then, we compared the strength of high-order interactive concepts between the standard DNN \(\mathbf{\psi}^{*}\) and the BNN \(\mathbf{\theta}_{\psi^{*}}\).

(3) We trained a standard DNN and a BNN with the same architecture. Then, we compared the strength of high-order interactive concepts between each pair of standard DNN \(\hat{\mathbf{\psi}}\) and the BNN \(\hat{\mathbf{\theta}}\) when these two networks were trained to have the same training accuracy. We used the training accuracy to align the learning progress of the two networks for fair comparison.

Specifically, the average strength of the \(s\)-order interactive concepts was measured as \(I_{\text{unegth}}^{(s)}=\mathbb{E}_{\mathbf{x}}[\mathbb{E}_{S\subseteq N,|S|=s}[l( S|\mathbf{x})]]\). To compute the interaction effect \(I(S|\mathbf{x})\), we set \(v(\mathbf{x}_{S})=\log\frac{p(y=y^{*}|\mathbf{x}_{S})}{1-p(y=y^{*}|\mathbf{x}_{S})}\in \mathbb{R}\), which reflected the confidence of classifying the masked input sample \(\mathbf{x}_{S}\) into the ground-truth category \(y^{*}\). For standard DNNs, \(p(y=y^{*}|\mathbf{x}_{S})\) referred to the classification probability of the ground-truth category on the masked sample \(\mathbf{x}_{S}\). For BNNs, \(p(y=y^{*}|\mathbf{x}_{S})\) was computed according to Eq. (2), where we sampled ten neural networks from the weight distribution \(q_{\mathbf{\theta}}(\mathbf{W})\) of the BNN, and computed the average classification probability over all these networks.

We followed experimental settings in the _experiments_ paragraph in Section 2.2 to train the networks. Specifically, we trained standard DNNs and BNNs with the MLP architecture on the TV news dataset, the Census dataset, and the MNIST dataset. We trained standard DNNs and BNNs with the LeNet architecture on the CIFAR-10 dataset. Appendix I introduces how to efficiently compute \(I(S|\mathbf{x})\) on images. Figure 4 shows that the strength of high-order interactive concepts of BNNs was much weaker than that of standard DNNs in all comparisons. This verified that BNNs were less likely to encode high-order (complex) interactive concepts than standard DNNs.

## 4 Conclusion and discussion

In this paper, we have proven the tendency of mean-field variational BNNs to avoid encoding high-order (complex) concepts. Many studies (Ren et al., 2021; Deng et al., 2021; Ren et al., 2023) have shown that there does exist a concept-emerging phenomenon when a neural network is sufficiently trained.

Besides, as discussed in the introduction, encoding less complex concepts does not mean that BNNs have weaker representation power than standard DNNs, because a standard DNN can be considered as a specific BNN with zero weight uncertainty. More crucially, Ren et al. (2021) and Lengerich et al. (2022) proved that high-order concepts are usually vulnerable to adversarial attacks and have weak generalization power. Thus, it is hard to say whether the tendency to avoid encoding complex concepts is a demerit or not.

Figure 4: (a) Comparison of the strength of interactive concepts (i) between a trained BNN \(\mathbf{\theta}^{*}\) and the constructed standard DNN \(\mathbf{\psi}_{\mathbf{\theta}^{*}}\), (ii) between a trained standard DNN \(\mathbf{\psi}^{*}\) and the constructed BNN \(\mathbf{\theta}_{\psi^{*}}\). (b) We trained a standard DNN \(\hat{\mathbf{\psi}}\) and a BNN \(\hat{\mathbf{\theta}}\) with the LeNet architecture on the CIFAR-10 dataset, and compared the strength of interactive concepts between the two networks when the two networks were trained to have the same training accuracy.