Masked Bayesian Neural Networks : Theoretical Guarantee and its
  Posterior Inference
Bayesian approaches for learning deep neural networks (BNN) have been
received much attention and successfully applied to various applications.
Particularly, BNNs have the merit of having better generalization ability as
well as better uncertainty quantification. For the success of BNN, search an
appropriate architecture of the neural networks is an important task, and
various algorithms to find good sparse neural networks have been proposed. In
this paper, we propose a new node-sparse BNN model which has good theoretical
properties and is computationally feasible. We prove that the posterior
concentration rate to the true model is near minimax optimal and adaptive to
the smoothness of the true model. In particular the adaptiveness is the first
of its kind for node-sparse BNNs. In addition, we develop a novel MCMC
algorithm which makes the Bayesian inference of the node-sparse BNN model
feasible in practice.
Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, Yongdai Kim
# Masked Bayesian Neural Networks :

###### Abstract

Bayesian approaches for learning deep neural networks (BNN) have been received much attention and successfully applied to various applications. Particularly, BNNs have the merit of having better generalization ability as well as better uncertainty quantification. For the success of BNN, search an appropriate architecture of the neural networks is an important task, and various algorithms to find good sparse neural networks have been proposed. In this paper, we propose a new node-sparse BNN model which has good theoretical properties and is computationally feasible. We prove that the posterior concentration rate to the true model is near minimax optimal and adaptive to the smoothness of the true model. In particular the adaptiveness is the first of its kind for node-sparse BNNs. In addition, we develop a novel MCMC algorithm which makes the Bayesian inference of the node-sparse BNN model feasible in practice.

Machine Learning, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference, Bayesian Inference Inference Inference, Bayesian Inference Inference Inference, Bayesian Inference
been suggested. A popular approach is to prune iteratively unnecessary edges (Han et al., 2015; Frankle and Carbin, 2018) or unnecessary nodes (Wen et al., 2016; He et al., 2017; Wang et al., 2019; Chin et al., 2020). However, such algorithms do not provide proper uncertainty quantification. By analyzing image data with CNNs, we illustrate that our node-sparse BNN is good at compressing DNNs without hampering the ability of uncertainty quantification.

Our contributions are summarized as follows:

* We develop a node-sparse prior for DNNs such that the posterior concentration rate to the true model is near minimax optimal adaptively to the smoothness of the true model and Bayesian inference with a specially designed MCMC algorithm is possible.
* We implement an efficient MCMC algorithm for searching good node-sparse BNNs. In particular, we develop a local informed proposal distribution in the Metropolis-Hastings (MH) algorithm to search good node-sparse architectures efficiently.
* By numerical experiments, we illustrate that the mBNN outperforms other Bayesian approaches including nonsparse BNN and VI algorithms in terms of generalization and uncertainty quantification.

## 2 Preliminaries

### Notation

Let \(\mathbb{R}\) and \(\mathbb{N}\) be the sets of real numbers and natural numbers, respectively. For an integer \(n\in\mathbb{N}\), we denote \([n]:=\{1,\ldots,n\}\). A capital letter denotes a random variable or matrix interchangeably whenever its meaning is clear, and a vector is denoted by a bold letter, e.g. \(\mathbf{x}:=(x_{1},\ldots,x_{d})^{\top}\). For a \(d\)-dimensional vector \(\mathbf{x}\in\mathbb{R}^{d}\), we denote \(|\mathbf{x}|_{p}:=(\sum_{j=1}^{d}|x_{j}|^{p})^{1/p}\) for \(1\leq p<\infty\), \(|\mathbf{x}|_{0}:=\sum_{j=1}^{d}\mathbb{I}(x_{j}\neq 0)\) and \(|\mathbf{x}|_{\infty}:=\max_{j\in[d]}|x_{j}|\). For a real-valued function \(f:\mathcal{X}\to\mathbb{R}\) and \(1\leq p<\infty\), we denote \(||f||_{p,n}:=(\sum_{i=1}^{n}f(\mathbf{x}_{i})^{p}/n)^{1/p}\) and \(||f||_{p,\mathbb{P}\mathbf{X}}:=\left(\int_{\mathbf{X}\in\mathcal{X}}f(\mathbf{X})^{p}d \mathbb{P}\mathbf{X}\right)^{1/p}\) where \(\mathrm{P}_{\mathbf{X}}\) is a probability measure defined on input space \(\mathcal{X}\). Moreover, we define \(||f||_{\infty}=\sup_{\mathbf{x}\in\mathcal{X}}|f(\mathbf{x})|\). For \(b_{1}\leq b_{2}\), we define \(f_{[b_{1},b_{2}]}(\cdot):=\min(\max(f(\cdot),b_{1}),b_{2})\) which is a truncated version of \(f\) on \(b_{1}\) and \(b_{2}\). We denote \(\circ\) and \(\odot\) as the composition of functions and element-wise product of vectors or matrices, respectively. For a probability vector \(\mathbf{q}\in\mathbb{R}^{r}\), we denote \(\mathrm{Cat}(\mathbf{q})\) as the categorical distribution with the probabilities of each category being \(\mathbf{q}\) and \(\mathrm{Multi}_{(N,\mathbf{q})}\) as the distribution of N many selected balls without replacement from the jar containing \(r\) many balls whose selection probabilities are \(\mathbf{q}\). For technical simplicity, we assume \(\mathcal{X}=[-1,1]^{d}\).

### Data generating process

We consider two supervised learning problems : regression and classification. In regression problems, the input vector \(\mathbf{X}\) and the response variable \(Y\in\mathbb{R}\) are generated from the model

\[\begin{split}\mathbf{X}\sim&\mathrm{P}_{\mathbf{X}},\\ Y|\mathbf{X}\sim& N(f_{0}(\mathbf{X}),\sigma_{0}^{2}), \end{split} \tag{1}\]

where \(\mathrm{P}_{\mathbf{X}}\) is the probability measure defined on \(\mathcal{X}\). Here, \(f_{0}:\mathcal{X}\to\mathbb{R}\) and \(\sigma_{0}^{2}>0\) are the unknown true regression function and unknown variance of the noise, respectively.

For \(K\)-class classification problems, the input vector \(\mathbf{X}\) and the response variable \(Y\in[K]\) are generated from the model

\[\begin{split}\mathbf{X}\sim&\mathrm{P}_{\mathbf{X}},\\ Y|\mathbf{X}\sim&\mathrm{Cat}\left(\mathrm{softmax}(\bm {f}_{0}(\mathbf{X}))\right),\end{split} \tag{2}\]

where \(\mathrm{P}_{\mathbf{X}}\) is the probability measure defined on \(\mathcal{X}\) and \(\mathbf{f}_{0}:\mathcal{X}\to\mathbb{R}^{K}\) is the logit of the unknown true conditional class probability function.

## 3 Masked Bayesian Neural Network

To construct a node-sparse BNN, we propose to use masking vectors which screen some nodes of the hidden layers. We first define the masked Deep Neural Network (mDNN) model, and then propose the mBNN on the top of the mDNN by specifying a prior appropriately.

### Deep Neural Network

For \(L\in\mathbb{N}\) and \(\mathbf{p}=(p^{(0)},p^{(1)},...,p^{(L)},p^{(L+1)})^{\top}\in\mathbb{N}^{L+2}\), DNN with the \((L,\mathbf{p})\) architecture is a DNN model which has \(L\) hidden layers and \(p^{(l)}\) many nodes at the \(l\)-th hidden layer for \(l\in[L]\). The input and output dimensions are \(p^{(0)}\) and \(p^{(L+1)}\), respectively. The output of the DNN model can be written as

\[f_{\mathbf{\theta}}^{\mathrm{DNN}}(\cdot):=A_{L+1}\circ\rho\circ A_{L}\cdots\circ \rho\circ A_{1}(\cdot), \tag{3}\]

where \(A_{l}:\mathbb{R}^{p^{(l-1)}}\mapsto\mathbb{R}^{p^{(l)}}\) for \(l\in[L+1]\) is an affine map defined as \(A_{l}(\mathbf{x}):=W_{l}\mathbf{x}+\mathbf{b}_{l}\) with \(W_{l}\in\mathbb{R}^{p^{(l)}\times p^{(l-1)}}\) and \(\mathbf{b}_{l}\in\mathbb{R}^{p^{(l)}}\) and \(\rho\) is the RELU activation function. The DNN model is parameterized by \(\mathbf{\theta}\) which is the concatenation of the weight matrices and bias vectors, that is

\[\mathbf{\theta}:=(\mathrm{vec}(W_{1})^{\top},b_{1}^{\top},\ldots,\mathrm{vec}(W_{L+ 1})^{\top},b_{L+1}^{\top})^{\top}.\]

### Masked Deep Neural Network

For a given standard DNN, the corresponding masked DNN (mDNN) is constructed by simply adding masking parameters to the standard DNN model. For \(l\in[L]\), the mDNN
model screens the nodes at the \(l\)-th hidden layer using the binary masking vector \(\mathbf{m}^{(l)}\in\{0,1\}^{p^{(l)}}\). When \(\left(\mathbf{m}^{(l)}\right)_{j}=0,\) the \(j\)-th node at the \(l\)-th hidden layer becomes inactive. The output of the mDNN model with the \((L,\mathbf{p})\) architecture can be written as

\[f_{\mathbf{M},\mathbf{\theta}}^{\mathrm{mDNN}}(\cdot):=A_{L+1}\circ\rho_{\mathbf{m}^{(L)}} \circ A_{L}\cdots\circ\rho_{\mathbf{m}^{(1)}}\circ A_{1}(\cdot),\]

where \(A_{l}\) for \(l\in[L+1]\) is the affine map defined in (3) and \(\rho_{\mathbf{m}^{(l)}}:\mathbb{R}^{p^{(l)}}\rightarrow\mathbb{R}^{p^{(l)}}\) for \(l\in[L]\) is the masked-RELU activation function defined as

\[\rho_{\mathbf{m}^{(l)}}\left(\begin{array}{c}x_{1}\\ \vdots\\ x_{p^{(l)}}\end{array}\right):=\mathbf{m}^{(l)}\odot\left(\begin{array}{c}\max (x_{1},0)\\ \vdots\\ \max(x_{p^{(l)}},0)\end{array}\right).\]

The model is parameterized by \(\mathbf{M}\) and \(\mathbf{\theta}\), where \(\mathbf{M}\in\{0,1\}^{\sum_{l=1}^{L}p^{(l)}}\) is the concatenate of the all masking vectors

\[\mathbf{M}:=\left(\mathbf{m}^{(1)}{}^{\top},\ldots,\mathbf{m}^{(L)}{}^{\top}\right)^{\top}\]

and \(\mathbf{\theta}\) is the concatenation of the weight matrices and the bias vectors in the standard DNN model.

Note that the mDNN is nothing but a standard DNN with the architecture \((L,\vec{\mathbf{p}})\) where \((\vec{\mathbf{p}})_{l}=\left|\mathbf{m}^{(l)}\right|_{0}.\) That is, the mDNN is a reparameterization of the standard DNN using the masking vectors. This reparameterization, however, allows us to develop an efficient MCMC algorithm, in particular for searching good architectures (i.e. good masking vectors).

### Prior and posterior distribution

We adopt a data-dependent prior \(\Pi_{n}\) on the parameters \(\mathbf{M}\) and \(\mathbf{\theta}\) (as well as \(\sigma^{2}\) for regression problems). We consider a data-dependent prior to ensure the optimal posterior concentration rate and adaptiveness. We assume that a priori \(\mathbf{M}\) and \(\mathbf{\theta}\) (as well as \(\sigma^{2}\) for regression problems) are independent.

For \(\mathbf{M}\), we use the following hierarchical prior. For each \(\mathbf{m}^{(l)}\) with \(l\in[L]\), let \(s^{(l)}:=|\mathbf{m}^{(l)}|_{0}\) be the sparsity of the masking vector \(\mathbf{m}^{(l)}\). We put a prior mass on \(s^{(l)}\) by

\[\Pi_{n}(s^{(l)})\propto e^{-(\lambda\log n)^{5}s^{(l)^{2}}}\qquad\text{ for }s^{(l)}\in[p^{(l)}], \tag{4}\]

where \(\lambda>0\) is a hyper-parameter. Note that the prior on \(s^{(l)}\) regularizes the width of the network, and more strong regularization is enforced as more data are accumulated. Given the sparsity level \(s^{(l)}\), the masking vector \(\mathbf{m}^{(l)}\) is sampled from the set \(\{\mathbf{m}^{(l)}\in\{0,1\}^{p^{(l)}}:|\mathbf{m}^{(l)}|_{0}=s^{(l)}\}\) uniformly. In other words,

\[\Pi_{n}(\mathbf{m}^{(l)}|s^{(l)})\overset{iid}{\propto}\frac{1}{\binom{p^{(l)}}{s ^{(l)}}}\mathbb{I}(|\mathbf{m}^{(l)}|_{0}=s^{(l)}) \tag{5}\]

and the prior of \(\mathbf{m}^{(l)}\) is the product of (4) and (5)

For the prior of \(\mathbf{\theta}\), we assume that

\[\theta_{i}\overset{iid}{\sim}\mathfrak{p}(\theta_{i})\qquad i\in[T]\,, \tag{6}\]

where \(T:=\sum_{l=0}^{L}(p^{(l)}+1)p^{(l+1)}\) is the length of the vector \(\mathbf{\theta}\), and choose \(\mathfrak{p}\) carefully to ensure desirable theoretical properties. For high-dimensional linear regression problems, Castillo & van der Vaart (2012) and Castillo et al. (2015) notice that using a heavy-tailed distribution for the prior of the regression coefficients is essential for theoretical optimality. Motivated by these observations, we consider a heavy-tailed distribution for \(\mathfrak{p}\). Let \(\mathfrak{P}\) be the class of polynomial tail distributions on \(\mathbb{R}\) defined as

\[\mathfrak{P}:=\left\{\mathfrak{p}:\lim_{x\rightarrow\infty}\frac{x^{-\log x }}{\mathfrak{p}(x)}\to 0\text{ and }\lim_{x\rightarrow\infty}\frac{x^{-\log x}}{\mathfrak{p}(-x)} \to 0\right\}.\]

Examples of polynomial tail distributions are the Cauchy distribution and Student's t-distribution. On the other hand, the Gaussian and Laplace distributions do not belong to \(\mathfrak{P}\). We assume that \(\mathfrak{p}\) belongs to \(\mathfrak{P}\). We will show in Section 4 that any prior in \(\mathfrak{P}\) yields the optimal posterior concentration rate.

For the prior of \(\sigma^{2}\) in regression problems, a standard distribution such as the inverse-gamma distribution can be used. Any distribution whose density is positive at the true \(\sigma_{0}^{2}\) works for theoretical optimality.

### Comparison with MC-dropout

The idea of masking nodes in DNN has been already used in various algorithms. Dropout (Srivastava et al., 2014) and MC-dropout (Gal & Ghahramani, 2016) are two representative examples, where they randomly mask the nodes of DNN during the training phase. While Dropout abolishes the masking vectors and uses the scaled-down version of the trained weights in the prediction phase, MC-dropout uses the trained weights obtained in the training phase multiplied by a random masking vectors. Since the masking vectors is treated as a random vectors following its posterior distribution at the prediction phase, our mBNN is similar to MC-dropout. A key difference between the mBNN and MC-dropout, however, is that the mBNN learns the distribution of the masking vectors from data via the posterior distribution but MC-dropout does not. That is, the mBNN learns the architecture of DNN from data, which makes the mBNN have good theoretical and empirical properties.

## 4 Theoretical optimalities

In this section, we derive the posterior concentration rates of the mBNNs for regression and classification problems, which are minimax optimal up to a logarithmic factor. In addition, we show that the mBNN achieves the optimal sparsity
asymptotically. We assume that \(\mathcal{D}^{(n)}:=\{(\mathbf{X}_{i},Y_{i})\}_{i\in[n]}\) are independent copies following the true distribution \(\mathbb{P}_{0}\) specified by either the model (1) or model (2).

### Posterior concentration rate for nonparametric regression

We consider the nonparametric regression model (1). We assume the true regression function \(f_{0}\) belongs to the \(\beta\)-Holder class \(\mathcal{H}_{d}^{\beta}\)1 Here, the \(\beta\)-Holder class \(\mathcal{H}_{d}^{\beta}\) is given as

Footnote 1: Theoretical results for hierarchical composition functions are provided in Appendix B.

\[\mathcal{H}_{d}^{\beta}:=\{f:[-1,1]^{d}\to\mathbb{R};||f||_{\mathcal{H}^{\beta} }<\infty\},\]

where \(||f||_{\mathcal{H}^{\beta}}\) denotes the Holder norm defined by

\[||f||_{\mathcal{H}^{\beta}} :=\sum_{\mathbf{\alpha}:|\mathbf{\alpha}|_{1}<\beta}\|\partial^{\mathbf{ \alpha}}f\|_{\infty}\] \[+\sum_{\mathbf{\alpha}:|\mathbf{\alpha}|_{1}=\lfloor\beta\rfloor}\sup_{ \begin{subarray}{c}\mathbf{x}_{1},\mathbf{x}_{2}\in[-1,1]^{d}\end{subarray}}\frac{| \partial^{\mathbf{\alpha}}f(\mathbf{x}_{1})-\partial^{\mathbf{\alpha}}f(\mathbf{x}_{2})|}{| \mathbf{x}_{1}-\mathbf{x}_{2}|_{\infty}^{\beta-\lfloor\beta\rfloor}}.\]

For inference, we consider the probabilistic model

\[Y_{i}\stackrel{{ ind.}}{{\sim}}N(f^{\rm mDNN}_{\mathbf{M},\mathbf{\theta} [-F,F]}(\mathbf{X}_{i}),\sigma^{2}),\]

where \(f^{\rm mDNN}_{\mathbf{M},\mathbf{\theta}}(\mathbf{X}_{i})\) has the \((L_{n},\mathbf{p}_{n})\) architecture with \(L_{n}\) and \(\mathbf{p}_{n}\) given as

\[L_{n}:= \left\lceil C_{L}\log n\right\rceil, \tag{7}\] \[p_{n}:= \left\lceil C_{p}\sqrt{n}\right\rceil,\] \[\mathbf{p_{n}}:= (d,p_{n},\dots,p_{n},1)^{\top}\in\mathbb{N}^{L_{n}+2} \tag{8}\]

for positive constants \(C_{L}\) and \(C_{p}\) that are defined in Lemma A.1. Then, the likelihood \(\mathcal{L}(w|\mathcal{D}^{(n)})\) of \(w:=(\mathbf{M},\mathbf{\theta},\sigma^{2})\) is expressed as

\[(2\pi\sigma^{2})^{-\frac{n}{2}}\exp\left(-\frac{\sum_{i=1}^{n}(Y_{i}-f^{\rm mDNN }_{\mathbf{M},\mathbf{\theta}\left\lfloor-F,F\right\rfloor}(\mathbf{X}_{i}))^{2}}{2\sigma ^{2}}\right),\]

and the corresponding posterior distribution is given as

\[\Pi_{n}(w\mid\mathcal{D}^{(n)})\propto\Pi_{n}(w)\mathcal{L}(w|\mathcal{D}^{(n )}),\]

where \(\Pi_{n}\) is the prior defined on Section 3.3.

In the following theorem, we show that the mBNN model achieves the optimal (up to a logarithmic factor) posterior concentration rate to the true regression function.

**Theorem 4.1** (Posterior Concentration of the mBNN for regression problems).: _Assume \(f_{0}\in\mathcal{H}_{d}^{\beta}\), \(\beta<d\), and there exist \(F>0\) and \(\sigma_{max}^{2}>0\) such that \(\|f_{0}\|_{\infty}\leq F\) and \(\sigma_{0}^{2}\leq\sigma_{max}^{2}\). Consider the mDNN model with the (\(L_{n},\mathbf{p}_{n}\)) architecture, where \(L_{n}\) and \(\mathbf{p}_{n}\) are given in (7) and (8). If we put the prior given as (4), (5) and (6) over \(\mathbf{M}\), \(\mathbf{\theta}\) and any prior on \(\sigma^{2}\) whose density (with respect to Lebesgue measure) is positive on its support \((0,\sigma_{max}^{2}]\), the posterior distribution concentrates to the \(f_{0}\) and \(\sigma_{0}^{2}\) at the rate \(\varepsilon_{n}=n^{-\beta/(2\beta+d)}\log^{\gamma}(n)\) for \(\gamma>\frac{5}{2}\) in the sense that_

\[\Pi_{n}\Big{(}(f,\sigma^{2}): ||f-f_{0}||_{2,\mathbb{P}_{X}}\] \[+|\sigma^{2}-\sigma_{0}^{2}|>M_{n}\varepsilon_{n}\Bigm{|}\mathcal{ D}^{(n)}\Big{)}\stackrel{{\mathbb{P}_{0}^{n}}}{{\to}}0\]

_as \(n\to\infty\) for any \(M_{n}\to\infty\), where \(\mathbb{P}_{0}^{n}\) is the probability measure of the training data \(\mathcal{D}^{(n)}\)._

The convergence rate \(n^{-\beta/(2\beta+d)}\) is known to be minimax lower bound when estimating the \(\beta\)-Holder smooth function (Tsybakov, 2009). Our concentration rate is near optimal up to a logarithmic factor and adaptive to the smoothness of the true model.

Comparison with other worksSimilar convergence rates are derived in the non-bayesian theoretical deep learning literature (Schmidt-Hieber, 2020; Kohler and Langer, 2021). However, the architectures considered in Schmidt-Hieber (2020) and Kohler and Langer (2021) depend on the smoothness \(\beta\) of the true regression function, which is rarely known in practice. In contrast, architecture and prior of the mBNN do not depend on the smoothness \(\beta\), which makes the mBNN very attractive.

Polson and Rockova (2018), Cherief-Abdellatif (2020) and Bai et al. (2020) also derive near optimal concentration rates for edge-sparse BNNs. Posterior computations for their smoothness-adaptive models, however, are almost impossible and inferential cost of edge-sparse BNNs could be large. Jantre et al. (2021) provides the posterior concentration rate for node-sparse BNN, but their result does not guarantee minimax optimality and is not adaptive to the smoothness of the true model. Theorem 4.1 is the first result for theoretical optimality of the Bayesian analysis for node-sparse DNNs.

### Posterior concentration rate for binary classification

Theoretical results for regression problem can be extended to classification problem. We consider the classification problem (2) with \(K=2\), and denote \(f_{0}:=\left(\mathbf{f}_{0}\right)_{2}-\left(\mathbf{f}_{0}\right)_{1}.\) We assume that \(f_{0}\) belongs to the \(\beta\)-Holder class \(\mathcal{H}_{d}^{\beta}\). For inference, we consider the probabilistic model

\[Y_{i}\stackrel{{ ind.}}{{\sim}}\mathrm{Bernoulli}(\phi\circ f^{\rm mDNN }_{\mathbf{M},\mathbf{\theta}[-F,F]}(\mathbf{X}_{i})),\]

where \(\phi\) is the sigmoid function and \(f^{\rm mDNN}_{\mathbf{M},\mathbf{\theta}}\) has the \((L_{n},\mathbf{p}_{n})\) architecture with \(L_{n}\) and \(\mathbf{p}_{n}\) given as (7) and (8). Then, the likelihood \(\mathcal{L}(w|\mathcal{D}^{(n)})\) of \(w:=(\mathbf{M},\mathbf{\theta})\) is ex
pressed as

\[\prod_{i=1}^{n}(\phi\circ f_{\mathbf{M},\mathbf{\theta}[-F,F]}^{\rm mDNN}(\mathbf{X}_{i}))^{Y_ {i}}(1-\phi\circ f_{\mathbf{M},\mathbf{\theta}[-F,F]}^{\rm mDNN}(\mathbf{X}_{i}))^{1-Y_{i}}.\]

In the following theorem, we prove that the mBNN model achieves the optimal (up to a logarithmic factor) posterior concentration rate to the true conditional class probability adaptive to the smoothness of the true model.

**Theorem 4.2** (Posterior Concentration of the mBNN for classification problems).: _Assume \(f_{0}\in\mathcal{H}_{d}^{\beta}\), \(\beta<d\), and there exists \(F>0\) such that \(\|f_{0}\|_{\infty}\leq F\). Consider the mDNN model with the (\(L_{n},\mathbf{p}_{n}\)) architecture where \(L_{n}\) and \(\mathbf{p}_{n}\) are given in (7) and (8). If we put the prior given as (4), (5) and (6) over \(\mathbf{M}\) and \(\mathbf{\theta}\), the posterior distribution concentrates to the true conditional class probability at the rate \(\varepsilon_{n}=n^{-\beta/(2\beta+d)}\log^{7}(n)\) for \(\gamma>\frac{5}{2}\) in the sense that_

\[\Pi_{n}\Big{(}f:||\phi\circ f-\phi\circ f_{0}||_{2,\mathbb{P}_{X}}>M_{n} \varepsilon_{n}\;\Big{|}\;\mathcal{D}^{(n)}\Big{)}\stackrel{{ \mathbb{P}_{0}^{n}}}{{\to}}0\]

_as \(n\to\infty\) for any \(M_{n}\to\infty\), where \(\mathbb{P}_{0}^{n}\) is the probability measure of the training data \(\mathcal{D}^{(n)}\)._

### Guaranteed sparsity of the mBNN

Not only the fast concentration rate, the mBNN achieves the optimal sparsity too. Theorem 4.3, which is a by-product of the proofs for Theorems 4.1 and 4.2, gives the level of sparsity of the mBNN.

**Theorem 4.3** (Guaranteed sparsity of the mBNN).: _Under the assumptions in Theorem 4.1 or Theorem 4.2, we have_

\[\Pi_{n}\Big{(}f:f=f_{\mathbf{M},\mathbf{\theta}[-F,F]}^{\rm mDNN},\;\max_{l\in[L]}|\mathbf{ m}^{(l)}|_{0}>\mathfrak{s}_{n}\;\Big{|}\;\mathcal{D}^{(n)}\Big{)}\stackrel{{ \mathbb{P}_{0}^{n}}}{{\to}}0\]

_as \(n\to\infty\), where \(\mathfrak{s}_{n}\) is defined as_

\[\mathfrak{s}_{n}:=\left\lceil C_{p}\left(n^{\frac{d}{2\beta+d}}(\log n)\right) ^{1/2}\right\rceil\]

_and \(\mathbb{P}_{0}^{n}\) is the probability measure of the training data \(\mathcal{D}^{(n)}\)._

Yarotsky (2017) proves that the lower bound of the sparsity of DNNs to approximate functions in \(\mathcal{H}_{d}^{\beta}\) is equal to \(\mathfrak{s}_{n}\) up to a logarithmic factor. That is, the mBNN automatically learns the optimal sparsity from data.

## 5 Posterior inference

Let \(\mathcal{D}^{(n)}:=\{(\mathbf{X}_{i},Y_{i})\}_{i\in[n]}\) be training data. Let \(w\) denote all of the parameters in the mBNN, that is, \(w:=(\mathbf{M},\mathbf{\theta},\sigma^{2})\) for the regression problem (1) and \(w:=(\mathbf{M},\mathbf{\theta})\) for the classification problem (2). For a new test example \(\mathbf{x}\in\mathcal{X}\), the prediction with the mBNN is done by the predictive distribution:

\[p(y\mid\mathbf{x},\mathcal{D}^{(n)})=\int_{w}p(y\mid\mathbf{x},w)\Pi_{n}(w\mid \mathcal{D}^{(n)})dw,\]

where \(\Pi_{n}(w\mid\mathcal{D}^{(n)})\) is the posterior posterior distribution of \(w\). When the integral is difficult to be evaluated, it is common to approximate it by the Monte Carlo method

\[p(y\mid\mathbf{x},\mathcal{D}^{(n)})\approx\frac{1}{T}\sum_{t=1}^{T}p(y\mid\mathbf{x},w^{(t)}),\]

where \(w^{(t)}\sim\Pi_{n}(w\mid\mathcal{D}^{(n)})\). In this section, we develop a MCMC algorithm to sample \(w\) efficiently from \(\Pi_{n}(w\mid\mathcal{D}^{(n)})\).

### MCMC algorithm

The proposed MCMC algorithm samples \(\mathbf{\theta}\) (and \(\sigma^{2}\)) given \(\mathbf{M}\) and \(\mathcal{D}^{(n)}\) and then samples \(\mathbf{M}\) given \(\mathbf{\theta}\) (and \(\sigma^{2}\)) and \(\mathcal{D}^{(n)}\), and iterates these two samplings until convergence.

There are various efficient sampling algorithms for \(\mathbf{\theta}\) (and \(\sigma^{2}\)) given \(\mathbf{M}\) and \(\mathcal{D}^{(n)}\) such as Hamiltonian Monte Carlo (HMC) (Neal et al., 2011), Stochastic Gradient Langevin Dynamics (SGLD) (Welling and Teh, 2011) and Stochastic Gradient HMC (SGHMC) (Chen et al., 2014). In practice, we select a sampling algorithm among those depending on the sizes of data and model.

For generating \(\mathbf{M}\) from its conditional posterior, we consider the Metropolis-Hastings (MH) algorithm. The hardest part is to design a good proposal distribution since the dimension of \(\mathbf{M}\) is quite large and all entries are binary. In the next subsection, we propose an efficient proposal distribution for \(\mathbf{M}\).

### Proposal for the MH algorithm

Essentially, sampling \(\mathbf{M}\) is equivalent to sampling a large dimensional binary vector. A well known strategy for sampling a large dimensional binary vector is to use the MH algorithm with the locally informed proposal (Umrigar, 1993; Zanella, 2020) given as

\[q\left(\mathbf{M}^{\star}\mid\mathbf{M}\right)\propto e^{\frac{1}{2}(l(\mathbf{M}^{\star})- l(\mathbf{M}))}\mathbb{I}\left(\mathbf{M}^{\star}\in H(\mathbf{M})\right), \tag{9}\]

where \(l(\mathbf{M})\) is the log-posterior of \(\mathbf{M}\) and \(H(\mathbf{M})\) is the Hamming ball of a certain size around \(\mathbf{M}\). The key point of (9) is to give more probability to \(\mathbf{M}^{\star}\) whose posterior probability is high. While powerful, this locally informed proposal requires to compute \(l(\mathbf{M}^{\star})\) for every \(\mathbf{M}^{\star}\in H(\mathbf{M})\), which is time consuming. To resolve this problem, we propose a proposal distribution which only uses the information of the current \(\mathbf{M}\).

First, we select either _birth_ or _death_, where _birth_ makes some inactive nodes become active and _death_ makes some active nodes become inactive. When _death_ is selected, motivated by the pruning algorithms (Lee et al., 2018; Tanaka et al., 2020), our strategy is to prune less sensitive nodes. That is, we delete nodes which do not affect much to the
current DNN model when their values are changed. On the other hand, when _birth_ is selected, we choose some of inactive nodes with equal probabilities and make them active. We use equal probabilities because the posterior of the edges connected to the inactive nodes is the same as the prior and thus there is no reason to prefer certain inactive nodes more. Moreover, the proposal with equal probabilities for _birth_ is helpful for increasing the acceptance rate of _death_ to result in fast mixing.

To be more specific, we first select the move \(u\in\{0,1\}\) with probability 1/2, where \(u=0\) and \(u=1\) indicates _birth_ and _death_, respectively. In addition, we select an integer \(N\) from \([N_{\max}]\) uniformly, where \(N_{\max}\) is a prespecified positive integer. Then, we select randomly \(N\) nodes among the nodes whose masking values are \(u\) following \(\mathrm{Multi}_{(N,\mathbf{Q}_{u})}\), where

\[\mathbf{Q}_{0}\propto\mathbb{I}(\mathbf{m}_{j}^{(l)}=0),\] (_birth_) (10) \[\mathbf{Q}_{1}\propto\mathbb{I}(\mathbf{m}_{j}^{(l)}=1)\exp\left(-|\nabla l (\mathbf{M})|/2\right).\] (_death_) (11)

Even though \(l(\mathbf{M})\) is defined only on binary vectors, the gradient \(\nabla l(\mathbf{M})=\partial l(\mathbf{M})/\partial\mathbf{M}\) of \(l(\mathbf{M})\) can be defined by extending the domain of \(l(\mathbf{M})\) appropriately. Finally, we flip the masking values of the selected nodes to have a new proposal \(\mathbf{M}^{*}\). To sum up, the proposal distribution first selects a set of nodes \(\{i_{1},\ldots,i_{N}\}\) following \(\mathrm{Multi}_{(N,\mathbf{Q}_{u})}\) and changes their mask values to \((1-\mathbf{M}_{i_{1}},\ldots,1-\mathbf{M}_{i_{N}})\). Then, we accept the new proposal \(\mathbf{M}^{*}\) with probability

\[\min\left(1,\frac{\Pi_{n}(\mathbf{M}^{*}\mid X^{(n)},Y^{(n)},\mathbf{\theta})}{\Pi_{n }(\mathbf{M}\mid X^{(n)},Y^{(n)},\mathbf{\theta})}\frac{q\left(\mathbf{M}\mid\mathbf{M}^{*} \right)}{q\left(\mathbf{M}^{*}\mid\mathbf{M}\right)}\right), \tag{12}\]

where

\[\frac{q\left(\mathbf{M}\mid\mathbf{M}^{*}\right)}{q\left(\mathbf{M}^{*}\mid\mathbf{M}\right)} =\frac{\mathrm{Multi}_{(N,\mathbf{Q}_{1-u})}(\{i_{1},\ldots,i_{N}\})}{\mathrm{Multi }_{(N,\mathbf{Q}_{u})}(\{i_{1},\ldots,i_{N}\})}\]

and \(\mathbf{Q}_{1-u}^{*}\) is the selection probability vector defined by (10) or (11) with the mask vectors \(\mathbf{M}^{*}\). Note that some inactive nodes become active when \(u=0\) (i.e. _birth_ of nodes) and some active nodes become inactive when \(u=1\) (i.e. _death_ of nodes). The proposed MH algorithm is summarized in Algorithm 1.

```
1: Sample \(u\sim\mathrm{Bernoulli}(0.5)\), \(N\sim\mathrm{Uniform}([N_{\max}])\).
2: Calculate the selection probability \(\mathbf{Q}_{u}\) using (10) or (11).
3: Sample \(N\) many nodes \(\{i_{1},\ldots,i_{N}\}\) by \[\{i_{1},\ldots,i_{N}\}\sim\mathrm{Multi}_{(N,\mathbf{Q}_{u})}.\]
4:\(\mathbf{M}^{*}=\mathrm{flip}(\mathbf{M}^{\text{curr}},\{i_{1},\ldots,i_{N}\})\), where \(\mathbf{M}^{\text{curr}}\) is the current masking vectors.
5: Accept or reject \(\mathbf{M}^{*}\) with the acceptance probability (12).
```

**Algorithm 1** The algorithm of the proposed MH algorithm

One may consider a linear approximation of \(l\) in (9) using the gradient information at \(\mathbf{M}\), as is suggested by Grathwohl et al. (2021) and Zhang et al. (2022). However, we found that the linear approximation is not accurate for the mBNN, which is partly because the corresponding DNN is not locally smooth enough. In section 6.5, we provide the results of the experiment for comparing several proposal distributions which support the choice of (10) and (11).

## 6 Experiment

In this section, we perform experiments to empirically justify the usefulness of the mBNN. In Section 6.1 and 6.2, we conduct an experiment to demonstrate the necessity of masking variables using simulation and real datasets. In Section 6.3, we apply the mBNN to a Bayesian structural time series model to illustrate its practical usefulness. In Section 6.4, we extend the mBNN to CNN and experimentally show it is also useful for compressing large complex DNNs. In Section 6.5, we investigate the efficiency of the proposal distribution (10) and (11) in the MH algorithm. All the experimental details as well as the results of additional numerical experiments are given in Appendix D and E. The code is available at [https://github.com/ggong369/mBNN](https://github.com/ggong369/mBNN).

### Simulation

We obtain the predictive distribution when the true regression function is the noisy polynomial regression problem considered in Hernandez-Lobato and Adams (2015). Inputs \(x_{i}\) are sampled from \(x_{i}\sim\mathrm{Uniform}(-4,4)\) and the corresponding outputs \(y_{i}\) are obtained by \(y_{i}=x_{i}^{3}+\epsilon_{i}\), where \(\epsilon_{i}\sim N(0,9)\). We generate 20 training examples and compare the predictive distribution of the mBNN with that of BNN by generating 1000 MCMC samples from each model. For each model, the two hidden layer MLP with the layer sizes (1000,1000) is used.

The mean and 95% predictive intervals of the predictive

Figure 1: **Simulated data.** Predictive distributions of BNN and the mBNN on simulated data. The true polynomial function and the noisy observations are plotted by the red line and black dots, respectively. For each predictive distribution, the mean is drawn by the blue line and the 95% predictive interval is shown as the shaded areas.
distributions of BNN and the mBNN are presented in Figure 1. The figure illustrates that BNN quantifies uncertainty on a data sparse region overly to have a too wide predictive interval. Note that the true regression function is assumed to smooth and hence it is possible to transfer information on data dense regions to data sparse regions. Thus, proper uncertainty quantification even on data sparse regions could be possible. The coverage probability of the predictive interval of the mBNN is 95.3%, which is obtained by generating additional 1000 test sample, while that of BNN is 98.6%. The results indicate that deleting unnecessary nodes is important not only for fast convergence rates but also proper uncertainty quantification.

### Real dataset

We evaluate BNN, node-sparse VI (NS-VI) (Louizos et al., 2017) and the mBNN on four UCI regression datasets (_Boston, Concrete, Energy, Yacht_). For each dataset, we construct 20 random 90-to-10 train-test splits to provide the standard errors. For each method, the two hidden layer MLP with the layer sizes (1000,1000) is used. We select 20 models from several thousands MCMC samples and compare the predictive distributions obtained by the selected 20 models in terms of generalization and uncertainty quantification.

In Table 1, we report the means and standard errors of the performance measures on the 20 repeated experiments. For the performance measures, the coverage probability of the 95% predictive interval (Coverage), the root mean square error (RMSE) of the Bayes estimator, the negative log-likelihood (NLL) and continuous ranked probability score (CRPS) (Gneiting and Raftery, 2007) on test data are considered.

It is obvious that the mBNN outperform the other two competitors with respect to all of the four measures. That is, the mBNN is good at not only estimating the regression function and but also uncertainty quantification. It is noticeable that NS-VI performs too badly, which suggests that full Bayesian analysis of DNN is must. More detailed results such as the sparsity and the sensitivity to the hyper-parameter selection are provided in Appendix E.

### Application to the Bayesian structural time series model

Bayesian structural time series (BSTS) models (Scott and Varian, 2014; Qiu et al., 2018) provide a useful tool for time series forecasting, nowcasting, inferring causal relationships and anomaly detection (Brodersen et al., 2015; Feng and Tian, 2021). The key of BSTS is the state space model, which is given as

\[y_{t} =\mu_{t}+\boldsymbol{\beta}^{\top}\boldsymbol{x}_{t}+\epsilon_{t}, \epsilon_{t} \sim\mathcal{N}(0,\sigma_{e}^{2})\] \[\mu_{t+1} =r\mu_{t}+\eta_{t}, \eta_{t} \sim\mathcal{N}(0,\sigma_{\eta}^{2})\] \[\mu_{0} \sim N(a_{0},\sigma_{0}^{2})\]

where \(\boldsymbol{x}_{t}\in\mathbb{R}^{d}\), \(y_{t}\in\mathbb{R}\) and \(\mu_{t}\in\mathbb{R}\) denote observed input variables, output variable and unobserved local trend at time \(t\), respectively. For inference and prediction, MCMC algorithm using Kalman filter (Welch et al., 1995; Durbin and Koopman, 2002) is mainly used.

In many cases, the linear component \(\boldsymbol{\beta}^{\top}\boldsymbol{x}_{t}\) may not be insufficient to explain complicate relations between inputs and outputs, and DNN can be considered instead. In turn, the mBNN is a useful inferential tool for this nonlinear BSTS. To illustrate that the mBNN works well for BSTS, we analyze a real dataset consisting of daily search volumes of several keywords collected by a Korea search platform company. The dataset consists of daily search volumes of keywords in year 2021 associated with a pre-specified product (eg. shampoo). The aim is to predict the search volume of the pre-specified product based on the search

\begin{table}
\begin{tabular}{c|c|c c c c} \hline
**Dataset** & **Method** & **Coverage** & **RMSE** & **NLL** & **CRPS** \\ \hline \multirow{3}{*}{Boston} & BNN & 0.912(0.000) & 3.411(0.145) & 2.726(0.087) & 1.738(0.052) \\  & NS-VI & 0.746(0.010) & 3.079(0.179) & 4.242(0.568) & 1.661(0.069) \\  & mBNN & **0.933(0.007)** & **2.092(0.143)** & **2.472(0.085)** & **1.462(0.055)** \\ \hline \multirow{3}{*}{Concrete} & BNN & 0.948(0.008) & 5.580(0.178) & 3.901(0.055) & 2.658(0.072) \\  & NS-VI & 0.568(0.011) & 5.0466(0.149) & 9.713(0.086) & 2.965(0.088) \\  & mBNN & **0.912**(0.009) & **4.913(0.180)** & **3.027**(0.046) & **2.628(0.087)** \\ \hline \multirow{3}{*}{Energy} & BNN & **0.948(0.045)** & 0.591(0.017) & 0.910(0.025) & 3.522(0.007) \\  & NS-VI & 0.913(0.006) & 1.322(0.117) & 1.792(0.121) & 0.720(0.055) \\ \cline{1-1}  & mBNN & **0.945**(0.004) & **0.744**(0.015) & **6.760**(0.034) & **0.256(0.006)** \\ \hline \multirow{3}{*}{Yacht} & BNN & 0.977(0.006) & 0.575(0.048) & 1.011(0.056) & 0.332(0.013) \\ \cline{1-1}  & NS-VI & 0.932(0.011) & 1.842(0.096) & 1.915(0.063) & 0.969(0.042) \\ \cline{1-1}  & mBNN & **0.953**(0.008) & **6.664**(0.044) & **0.932**(0.062) & **0.318**(0.015) \\ \hline \end{tabular}
\end{table}
Table 1: **Real dataset.** Performance comparison of BNN, NS-VI and the mBNN on UCI datasets. The boldface numbers are the best one among the three methods.

Figure 2: **Daily search volume dataset.** Predictive distribution of BSTS model using linear, DNN and mDNN. X-axis and y-axis represent time (in day) and search volumes (in thousand), respectively. For each method, the means of predictive distributions and 95% predictive intervals are shown as the blue line and shaded area, respectively. The observed values are plotted by the red line.
volumes of other related keywords.

We apply the three BSTS models corresponding to the three regression components - linear, BNN and the mBNN. For BNN and the mBNN, the two hidden layer MLP with the layer sizes (100,100) is used. The predictive intervals with the NLL and RMSE values on the data from \(t=241\) to \(t=365\) obtained by the posterior distribution inferred on the data from \(t=1\) to \(t=240\) are presented in figure 2. It is clearly observed that the mBNN is superior in both nowcasting ability and uncertainty quantification. It is interesting that the predictive intervals of the linear and BNN models are much wider than those of the mBNN, which amply indicates that the choice of an appropriate architecture of DNN is crucial for desirable uncertainty quantification. More details about the dataset, methodology and additional experimental results are provided in Appendix D.3.

### Extension to convolution neural network

We also extend the mBNN to masked Bayesian convolution neural network (mBCNN) for image dataset. We construct a masked CNN (mCNN) by adding masking vectors to the CNN model. See Appendix C.2 for details of the mBCNN.

We evaluate compression ability of the mBCNN on image datasets. For comparison, we use node-sparse versions of approximated Bayesian methodologies : NS-VI, node-sparse Deep ensemble (NS-Ens) and node-sparse MC-dropout (NS-MC). Detail description of competitors are provided in Appendix D.4. For each methods, ResNet18 (He et al., 2016) architecture is used. For fair comparison, we use five networks for inference in all the methods. We repeat the experiments 3 times with different seeds.

In Table 2, we report the means and standard errors of the performance measures on the repeated experiments. For the performance measures, accuracy of the Bayes estimator, NLL, expected calibration error (ECE) (Kumar et al., 2019) of test data are considered. In addition, we report inferential costs (FLOPs) and the numbers of nonzero parameters (Capacity) relative to the non-sparse model. The results in Table 2 show that mBCNN provides better generalization and uncertainty quantification with less inferential cost and model capacity compared to the other competitors. That is, the mBNN is a useful tool for compressing complex DNNs without hampering uncertainty quantification much.

### Ablation study: Efficiency of the proposal

To illustrate the efficiency of our proposal distribution in Algorithm 1, we conduct a comparative experiment. As an alternative to the proposal distribution with selection probability (10) for _birth_ and (11) for _death_, we consider the following candidates for the selection probability of either _birth_ or _death_:

\[\mathbf{Q}_{u} \propto\mathbb{I}(\mathbf{m}_{j}^{(l)}=u), \tag{13}\] \[\mathbf{Q}_{u} \propto\mathbb{I}(\mathbf{m}_{j}^{(l)}=u)\exp\left(-|\nabla l(\mathbf{M}) |/2\right),\] (14) \[\mathbf{Q}_{u} \propto\mathbb{I}(\mathbf{m}_{j}^{(l)}=u)\exp\left((1-2u)\nabla l(\bm {M})/2\right). \tag{15}\]

While (13) gives the same selection probability on each active (or inactive) node, (14) and (15) put proposals depending on the role of each node. The proposal of (14) is devised to select less sensitive nodes more by giving proposal probabilities reciprocally proportional to \(|\nabla l(\mathbf{M})|\), and (15) uses a linear approximation of \(l\) in (9) (Grathwohl et al., 2021).

We compare the speeds of convergence of the MCMC algorithm with the 9 proposal distributions which are all \(3^{2}\) combinations of the three candidates (13), (14) and (15) and the two moves for _birth_ and _death_. Figure 3 presents how the ratios of the activated nodes relative to the largest DNN decrease in the burn-in phase of the MCMC algorithm. It is obvious that our proposal (the red solid lines) is most fast to eliminate unnecessary nodes. Note that the proposals involving the linear approximation of \(l(\mathbf{M})\) do not work well, which suggests that DNNs are not smooth enough to be approximated linearly.

\begin{table}
\begin{tabular}{c|c|c c c c} \hline  & **Measure** & **NS-VI** & **NS-Ens** & **NS-MC** & **mBCNN** \\ \hline \multirow{4}{*}{**mBCNN**} & ACC & 0.9606(0.002) & 0.926(0.003) & 0.918(0.002) & **0.932(0.001)** \\  & NLL & 0.298(0.004) & 0.255(0.011) & 0.525(0.006) & **0.220**(0.005) \\  & ECE & 0.009(0.001) & 0.012(0.002) & 0.020(0.001) & **0.088**(0.001) \\  & FLOPs & 4.485(0.076) & 22.45(1.647) & 41.16(0.009) & **12.26**(0.06)\% \\  & Capacity & 23.24(1.56\%) & 12.78(0.75)\% & 41.07(0.00\%) & **3.47**(0.03)\% \\ \hline \multirow{4}{*}{**mBCNN**} & ACC & 0.600(0.004) & 0.735(0.003) & 0.679(0.002) & **0.737**(0.001) \\  & NLL & 1.977(0.039) & 1.076(0.014) & 2.792(0.061) & **1.064**(0.008) \\ \cline{1-1}  & ECE & 0.006(0.000) & **0.002**(0.000) & 0.006(0.000) & **0.002**(0.000) \\ \cline{1-1}  & FLOPs & 37.38(0.61\%) & 27.02(2.30\%) & 53.50(0.00\%) & **18.34**(0.27)\% \\ \cline{1-1}  & Capacity & 41.61(0.49\%) & 20.08(1.15\%) & 53.45(0.00\%) & **12.21**(0.07)\% \\ \hline \end{tabular}
\end{table}
Table 2: **Image datasets.** Performance comparison of NS-VI, NS-Ens, NS-MC and mBCNN on image datasets. When inferring the posterior of mBCNN, we use SGLD (Welling & Teh, 2011).

Figure 3: **Efficiency of the proposal.** The ratios of activated nodes (y-axis) relative to the largest DNN are drawn as the MCMC algorithms iterate in the burn-in period (x-axis). The solid, dashed and dotted lines correspond to (13), (14) and (15) for _birth_, respectively while blue, red and green lines correspond to (13), (14) and (15) for _death_, respectively. It is obvious that the number of activated nodes with our proposal distribution (the solid red line) decreases much faster than the other proposals.
## 7 Discussion

We have proposed the mBNN which searches for a DNN with an appropriate complexity. We prove theoretical optimalities of the mBNN and develop an efficient MCMC algorithm. By extensive numerical studies, we illustrate that the proposed BNN discovers well condensed DNN architectures with better prediction accuracy and uncertainty quantification compared to large DNNs.

A node-sparse network can be considered as a dense network with a smaller width because all edges connected to survived nodes are active. This property is sharply contrast with edge-sparse networks. A key difference of the node-sparse network and a dense network with a smaller width is that the widths of each layer can vary for node-sparse networks while they should be fixed in advance for dense network. In practice, it would be difficult to find the optimal width in advance before analyzing data. mBNN is a kind of tools to find the optimal width data adaptively.

We do not insist that the proposal distribution in our MCMC algorithm is optimal. There would be more efficient proposals. More data adaptive proposals for _birth_ would be possible which we leave as a future work.

Condensing the posterior distribution is also interesting. We have to employ multiple DNNs in the prediction phase, which would be expensive. Summarizing multiple posterior samples into a single random DNN would be useful. For bootstrapping, Shin et al. (2021) proposes a similar method, which can be modified for the mBNN.

## Acknowledgements

This work was supported by National Research Foundation of Korea(NRF) grant funded by the Korea government (MSIT) (No. 2020R1A2C3A0100355014), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics ] and INHA UNIVERSITY Research Grant.
