Free energy of Bayesian Convolutional Neural Network with Skip
  Connection
Since the success of Residual Network(ResNet), many of architectures of
Convolutional Neural Networks(CNNs) have adopted skip connection. While the
generalization performance of CNN with skip connection has been explained
within the framework of Ensemble Learning, the dependency on the number of
parameters have not been revealed. In this paper, we show that Bayesian free
energy of Convolutional Neural Network both with and without skip connection in
Bayesian learning. The upper bound of free energy of Bayesian CNN with skip
connection does not depend on the oveparametrization and, the generalization
error of Bayesian CNN has similar property.
Shuya Nagayasu, Sumio Watanabe
# Free energy of Bayesian Convolutional Neural Network with Skip Connection

###### Abstract

Since the success of Residual Network(ResNet), many of architectures of Convolutional Neural Networks(CNNs) have adopted skip connection. While the generalization performance of CNN with skip connection has been explained within the framework of Ensemble Learning, the dependency on the number of parameters have not been revealed. In this paper, we show that Bayesian free energy of Convolutional Neural Network both with and without skip connection in Bayesian learning. The upper bound of free energy of Bayesian CNN with skip connection does not depend on the overparametrization and, the generalization error of Bayesian CNN has similar property.

**Keywords.**_Learning theory; Convolutional Neural Network; Bayesian Learning; Free Energy_

## 1 Introduction

Convolutional Neural Networks (CNNs) are a type of Neural Networks mainly used for computer vision. CNNs have been shown the high performance with deep layers [1, 2]. Residual Network(ResNet) [3] adopted the skip connection for addressing the problem that the loss function of CNN with deep layers does not decrease well through optimization. After success of ResNet, the CNNs with more than 100 layers are realized. The high performance of ResNet has been explained by similarity to the ensemble learning [4, 5, 6]. On the other hand, there is a common issue in neural networks that the reason why the overparametrized deep neural network generalized has been unknown yet.

In conventional learning theory, if the Fisher information matrix of a learning machine is positive definite, and the data size is sufficient large, the generalization error of the learning machine is determined from the number of its parameter in maximum likelihood estimator [7]. The similar property is shown in free energy and generalization error in Bayesian learning [8, 9, 10]. From these characteristics of generalization error and free energy some information criteria such as AIC, BIC, MDL are proposed. However, most of the hierarchical models such as neural networks have degenerated Fisher information matrix. In such models, the Bayesian generalization error and free energy are determined by a rational number called Real Log Canonical Threshold(RLCT) and that is smaller than the number of parameters [11, 12]. In particular, RLCTs are revealed in some concrete models such as three layered
neural networks [13, 14], normal mixtures [15, 16], Poisson mixtures [17], Boltzmann machine [18, 19], reduced rank regression [20], Latent Dirichlet allocation [21], matrix factorization, and Bayesian Network [22]. While RLCTs of many hierarchical models are revealed, that of neural networks with multiple layer of nonlinear transformation has not been clarified. Yet the possibility of that is shown in [23], the RLCT of Deep Neural Network is revealed [24]. On the other hand the RLCT of neural networks other than DNN was not explored.

In Bayesian learning for neural networks, how to realize the posterior is important. There exist approaches for generating posterior, Variational Approximation or Markov chain Monte Carlo(MCMC) methods. Variational Approximation for neural netowrks, Variational Autoencoder [25] or Monte Carlo dropout [26] are practically used. Also for CNNs, variational approach for Bayesian inference was proposed [27]. MCMC for neural networks, Hamiltonian Monte Carlo or Langevin Dynamics are useful for sampling from posterior. Stochastic Gradient Langevin Dynamics(SGLD) [28] is a MCMC method applying Stochastic Gradient Descent instead of Gradient Descent to Langevin Dynamics is popular MCMC for Bayesian Neural Networks. [29] used SGLD for generating posterior of CNNs.

In this paper we clarify the free energy and generalization error of Bayesian CNNs with and without skip connection. In both case the free energy and generalization error don't depend on the number of parameters in redundant filters. Then, in case with skip connection, the redundant layers affect the free energy and generalization error whereas they don't affect in case without skip connection. This paper consists of seven main sections and one appendix. In section2, we describe the setting of Convolutional Neural Network analyzed in this paper. In section3, we explain the basic terms of the Bayesian learning. In section4, we note the main theorem of this paper. In section5, we conducts the experiment of synthetic data. In section6 and section7, we discuss about the theorem in this paper and conclusion. In appendixA, we prove the main theorem of this paper.

## 2 Convolutional Neural Network

In this section we describe the function of Convolutional Neural Network. First, we explain CNN without skip connection. The kernel size is \(3\times 3\) with zero padding and 1-stride. The activation function is ReLU. The numbers of the layers of the CNN are \(K_{1}(\geq 3)\) for Convolutional Layers and \(K_{2}(\geq 3)\) for Fully Connected Layers.

Let \(x\in\mathbb{R}^{L_{1}\times L_{2}\times H_{1}}\) be an input vector generated from \(q(x)\) with bounded support and \(y\in\mathbb{N}\) be an output vector with \(q(y|x)\) whose support is \(\{1,\dots,H_{K_{1}+K_{2}}\}\). We define \(w^{(k)}\in\mathbb{R}^{3\times 3\times H_{k-1}\times H_{k}}\), \(b^{(k)}\in\mathbb{R}^{H_{k}}\) as weight and bias parameters in each Convolutional Layer (\(2\leq k\leq K_{1}\)). \(f^{(k)}\in\mathbb{R}^{L_{1}\times L_{2}\times H_{k}}\) is output of each layer for \(1\leq k\leq K_{1}\). \(\text{Conv}(f,w)\) is the convolution operation with zero padding and 1-stride:

\[\text{Conv}(f^{k-1},w^{k})_{l_{1},l_{2},h_{k}}=\sum_{h_{k-1}}\sum_{p=1,q=1}^{ p=3,q=3}f_{l_{1}+p-1,l_{2}+p-1,h_{k-1}}w_{p,q,h_{k-1},h_{k}}. \tag{1}\]

We define \(g(b^{k}):\mathbb{R}^{H_{k}}\rightarrow\mathbb{R}^{L_{1}\times L_{2}\times H_{ k}}\) as

\[g(b^{(k)})_{l_{1},l_{2}}=b^{(k)} \tag{2}\]
for \(1\leq l_{1}\leq L_{1},1\leq l_{2}\leq L_{2}\). By using \(w^{(k)}\), \(g(b^{(k)})\), and \(f^{(k-1)}\), \(f^{(k)}\) is described by

\[f^{(k)}(w,b,x)=\sigma(\text{Conv}(f^{(k-1)}(w,b,x),w^{(k)})+g(b^{(k)})) \tag{3}\]

where \(w,b\) are the set of all weight and bias parameters. \(\sigma()\) is a function that applies the ReLU to all the elements of the input tensor.

The output of \(k=K_{1}+1\) layer is result of Global Average Pooling on \(k=K_{1}\) layer:

\[f^{(K_{1}+1)}(w,b,x)=\frac{1}{L_{1}L_{2}}\sum_{l_{1}=1}^{l_{1}=L_{1}}\sum_{l_{ 2}=1}^{l_{2}=L_{2}}f^{(K_{1})}(w,b,x)_{l_{1},l_{2}}. \tag{4}\]

Let \(w^{(k)}\in\mathbb{R}^{H_{k}}\times\mathbb{R}^{H_{k-1}}\), \(b^{(k)}\in\mathbb{R}^{H_{k}}\) be weight and bias parameters in each Fully Connected Layer (\(K_{1}+2\leq k\leq K_{1}+K_{2}\)). For \(K_{1}+2\leq k\leq K_{1}+K_{2}-1\), \(f^{(k)}\) is defined by

\[f^{(k)}(w,b,x)=\sigma(w^{(k)}f^{(k-1)}(w,b,x)+b^{(k)}), \tag{5}\]

and for \(k=K_{1}+K_{2}\),

\[f^{(K_{1}+K_{2})}(w,b,x)=\text{softmax}(w^{(k)}f^{(k-1)}(w,b,x)+b^{(k)}), \tag{6}\]

where \(\text{softmax}()\) is a softmax function

\[\text{softmax}(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{J}e^{z_{j}}}. \tag{7}\]

The output of the model is represented stochastically

\[y\sim\text{Categorical}(f^{(K_{1}+K_{2})}(w,b,x)) \tag{8}\]

where Categorical() is a categorical distribution.

Then we describe CNN with skip connection. The number of layers within the skip connection is \(K_{s}\) and the number of skip connection is \(M\). The output of the layer with skipped connection is described by

\[f^{(mK_{s}+2)}(w,b,x)=\sigma(\text{Conv}(f^{(mK_{s}+1)}(w,b,x),w ^{(mK_{s}+2)})\] \[+B^{(mK_{s}+2)}+f^{((m-1)K_{s}+2)}(w,b,x)). \tag{9}\]

In this case, CNN satisfies the following conditions

\[K_{1} =MK_{s}+2\] \[H^{mK_{s}+2} =\text{const}(1\leq m\leq M). \tag{10}\]

The other conditions are the same as the case without skip connection.

Figure1 shows the configuration of Convolutional Neural Network analyzed in this paper.
## 3 Free energy in Bayesian Learning

### Bayesian Learning

Let \(X^{n}=(X_{1},\cdots X_{n})\) and \(Y^{n}=(Y_{1},\cdots Y_{n})\) be training data and labels. \(n\) is the number of the data. These data and labels are generated from a true distribution \(q(x,y)=q(y|x)q(x)\). The prior distribution \(\varphi(w)\), the learning model \(p(y|x,w)\) is given on the bounded parameter set \(W\). Then the posterior distribution is defined by

\[p(w|X^{n},Y^{n})=\frac{1}{Z(Y^{n}|X^{n})}\varphi(w)\prod_{i=1}^{n}p(Y_{i}|X_{i},w) \tag{11}\]

where \(Z_{n}=Z(Y^{n}|X^{n})\) is normalizing constant denoted as marginal likelihood:

\[Z_{n}=\int\varphi(w)\prod_{i=1}^{n}p(Y_{i}|X_{i},w)\mathrm{d}w. \tag{12}\]

The free energy is negative log value of marginal likelihood

\[F_{n}=-\log Z_{n}. \tag{13}\]

Free energy is equivalent to evidence and stochastic complexity.

The posterior predictive distribution is defined as the average of the model by posterior:

\[p^{*}(y|x)=p(y|x,X^{n},Y^{n})=\int p(y|x,w)p(w|X^{n},Y^{n})\mathrm{d}w. \tag{14}\]

Generalization error \(G_{n}\) is given by Kullback-Leibler divergence between the true distribution and posterior distribution as follows

\[G_{n}=\int q(y|x)q(x)\log\frac{q(y|x)}{p^{*}(y|x)}\mathrm{d}x\mathrm{d}y. \tag{15}\]

Average of Generalization error is difference between the average of Free energy of \(n\) and \(n+1\):

\[\mathbb{E}[G_{n}]-S=\mathbb{E}[F_{n+1}]-\mathbb{E}[F_{n}], \tag{16}\]

where \(\mathbb{E}[f(X^{n},Y^{n})]\) is the average of the generation of n data \(\mathbb{E}_{X^{n},Y^{n}}[f(X^{n},Y^{n})]\).

Figure 1: The structure of Convolutional Neural Network with and without Skip Connection
### Asymptotic property of Free energy and Generalization error

It is well known that if the average Kullback-Leibler divergence

\[K(w)=\int q(y|x)q(x)\log\frac{q(y|x)}{p(y|x,w)}\mathrm{d}x\mathrm{d}y. \tag{17}\]

can be approximated by quadratic form, in other words, the Laplace approximation can be applied to the posterior distribution, average of Free energy has the following asymptotic expansion with the number of parameters of learning model\(d\)[8, 9]

\[E[F_{n}]=n(S+\mathrm{Bias})+\frac{d}{2}\log n+O(1) \tag{18}\]

where \(S\) is entropy of true distribution and Bias is the minimum value of \(K(w)\) for \(w\in W\). The generalization error is calculated from Free energy by using equation(16) [10]:

\[E[G_{n}]=\mathrm{Bias}+\frac{d}{2n}+o\left(\frac{1}{n}\right). \tag{19}\]

Laplace approximation cannot be applied to the average Kullback-Leibler divergence of hierarchical model such as Gaussian Mixture or neural networks because of the degeneration of Fisher information matrix. In such models, the average of Free energy and Generalization error have the following asymptotic expansions [11]

\[E[F_{n}] =n(S+\mathrm{Bias})+\lambda\log n+o(\log n), \tag{20}\] \[E[G_{n}] =\mathrm{Bias}+\frac{\lambda}{n}+o\left(\frac{1}{n}\right), \tag{21}\]

where \(\lambda\) is a rational number called Real Log Canonical Threshold(RLCT). In particular, [24] showed that in case \(\mathrm{Bias}=0\) and \(x\) is bounded, when the Deep Neural Network is trained from the data generated from smaller network,

\[\lambda\leq\frac{d^{*}}{2} \tag{22}\]

where \(d^{*}\leq d\) is the number of parameter of data generating Network.

## 4 Main Theorem

In this subsection the main result of this paper is introduced. First, to state the theorem, we define the data generating network. Both in skip connection the data generating network satisfies the following conditions about the number of layers and filters,

\[K_{1}^{*}\leq K_{1},K_{2}^{*}\leq K_{2},(H^{*})^{(1)}=H^{(1)}(H^{*})^{(K_{1}) }=H^{(K_{1}+K_{2})} \tag{23}\]

and

\[H^{(k)} \geq(H^{*})^{(K_{1}^{*})}(K_{1}^{*}+1\leq k\leq K_{1})\] \[H^{(k)} \geq(H^{*})^{(K_{1}+K_{2}^{*})}(K_{1}+K_{2}^{*}+1\leq k\leq K_{1} +K_{2}-1)\] \[H^{(k)} \geq(H^{*})^{(k)}(others). \tag{24}\]

Then, we show the main theorem.
**Theorem 4.1**.: _(No Skip connection) Assume that the learning machine and the data generating distribution are given by \(p(y|x,w,b)\) and \(q(y|x)=p(y|x,w^{*},b^{*})\) in case without skip connection which satisfy the conditions (23) and (24), and that a training data \(\{(X_{i},Y_{i})\;\;i=1,2,...,n\}\) is independently taken from \(q(x)q(y|x)\). Then the average free energy satisfies the inequality,_

\[\mathbb{E}[F_{n}]\leq nS+\lambda_{CNN}\log n+C \tag{25}\]

_where_

\[\lambda_{CNN}=\frac{1}{2}\left(|w^{*}|_{0}+|b^{*}|_{0}+\sum_{k=K_{1}^{*}+1}^{ K_{1}}(9H_{K_{1}^{*}}+1)H_{K_{1}^{*}}\right) \tag{26}\]

_where \(|w^{*}|_{0},|b^{*}|_{0}\) are the numbers of parameters of weights and biases in data generating network._

**Theorem 4.2**.: _(Skip connection) Assume that the learning machine and the data generating distribution are given by \(p(y|x,w,b)\) and \(q(y|x)=p(y|x,w^{*},b^{*})\) in case with skip connection which satisfy the conditions (10), (23) and (24), and that a training data \(\{(X_{i},Y_{i})\;\;i=1,2,...,n\}\) is independently taken from \(q(x)q(y|x)\). Then_

\[\lambda_{CNN}=\frac{1}{2}(|w^{*}|_{0}+|b^{*}|_{0}) \tag{27}\]

Proof of main theorems are shown in AppendixA.

If there exists asymptotic expansion of the generalization error \(\mathbb{E}[G_{n}]\) in theorem4.1 and theorem4.2, that satisfies the following inequality

\[\mathbb{E}[G_{n}]\leq\frac{\lambda_{CNN}}{n}+o\left(\frac{1}{n}\right), \tag{28}\]

where

\[G_{n}=\int q(x)\sum_{i=1}^{H_{K_{1}+K_{2}}}f_{i}^{(K_{1}^{*}+K_{2}^{*})}(w^{*},b^{*},x)\log\frac{f_{i}^{(K_{1}^{*}+K_{2}^{*})}(w^{*},b^{*},x)}{\mathbb{E}_{w,b}[f_{i}^{(K_{1}+K_{2})}(w,b,x)]}\mathrm{d}x \tag{29}\]

which corresponds to categorical cross entropy.

## 5 Experiment

In this section, we show the result of experiment of synthetic data.

### Methods

We prepared the 2-class labeled simple data shown in fig2. The the data is \(x\in R^{4\times 4}\) and the values of each elements are in \((-1,1)\). The average of each element is \(0.5\) or \(-0.5\) and added the truncated normal distribution noise within the interval\((-0.5,0.5)\). The probability of each label of data is \(0.5\). We trained CNN whose number of convolutional layer \(K_{1}=2\) and
fully connected layers\(K_{2}=2\) with SGD. The number of filter is \(H_{2}=2\) and the parameters are \(L_{2}\) regularized. We use the trained CNN named "true model" as a data generating distribution. Note that the label of original data fig2 is deterministic but the label of true model is probabilistic. We prepare three learning CNN models. Each number of convolutional layers is \(K_{1}=2,3,4\). Each model has skip connection every one layers or does not have skip connection. The number of filters in each layers is \(H^{(k)}=4\). They have \(K_{2}=2\) fully connected layers. The prior distribution is the Gaussian distribution which covariance matrix is \(10^{4}I\) for weight parameter and \(10^{2}I\) for bias parameter. We train the learning CNN models by using the Langevin dynamics. The learning rate is \(10^{-2}\) and the interval of sampling is \(100\). We use the average of \(1000\) samples of learning CNN models as the average of posterior. We estimate the generalization error by the test error of \(10000\) test data from true model. We trained each learning model \(10\) times and estimated the \(\mathbb{E}[G_{n}]\) from the average of test error.

### Result of experiments

Table1 shows the result of the experiment. Test Error shows n times of the average of \(10\) test error in each model and the standard error of them. \(d_{\text{model}}\) is a number of parameters of each model. All the CNN models include the true model, hence the bias is \(0\). Then from equation(21), theoretical upper bound of the generalization error is \(\lambda_{\text{CNN}}/n\). In table1, the experimental values of all models are smaller than \(d_{\text{model}}/2\). Moreover in case with skip connection, the experimental value did not so increase with the increase of the number of layer. Then, in case \(K_{1}=4\) without skip connection, the experimental value increased from

\begin{table}
\begin{tabular}{c c c c} \hline model & \(n\times\) Test Error & \(\lambda_{\text{CNN}}\) & \(d_{\text{model}}/2\) \\ \hline \hline \(K_{1}=2\) & 16.0(1.9) & 13 & 25 \\ \(K_{1}=3\) no skip & 10.0(0.9) & 32 & 99 \\ \(K_{1}=4\) no skip & 58.4(2.3) & 51 & 173 \\ \(K_{1}=3\) with skip & 11.4(2.3) & 13 & 99 \\ \(K_{1}=4\) with skip & 15.6(1.2) & 13 & 173 \\ \hline \end{tabular}
\end{table}
Table 1: Experimental value and theoretical upper bound of the generalization error

Figure 2: The average of input x of each label
the case\(K_{1}=2\). In case \(K_{1}=3\) without skip connection, the experimental value is smaller than that of \(K_{1}=2\). Behavior of MCMC is considered to be the cause of this result. Since MCMC in high dimensional model needs the long series for convergence in general, the result is deviated from theoretical predict.

## 6 Discussion

### Difference with or without Skip Connection

In this paper for analyzing the overparametrized CNN, the data generating network is smaller than learning network both case of Skip Connection. Nevertheless two cases of the data generating network is different, if the learning model network has double filter \(H^{(k)}\) to the data generating network in each Convolutional Layer, the model network can represent the generating network in different case. The output of each layer is nonnegative hence the model can represent the skip connection or the negative of that. If the model network doesn't have larger layer to the data generating network, the free energy of CNN with skip connection can be both larger or smaller than that without skip connection by the data generating network. Then, the layer of model network gets larger, the free energy of CNN with skip connection does not change but that without skip connection gets larger and the free energy of CNN with skip connection comes to have smaller free energy for all data generating network.

### Comparison to Deep Neural Network

Firstly we compare the result of this paper to that of DNN in [24]. In case of DNN, the free energy depends on the layers of the model and only on that of the data generating network. This stands to the reason that mapping of the linear transformation in lower layer can be represented in higher layer. On the other hand, convolution operation doesn't have such property hence, the free energy of CNN without skip connection depends on the layer of learning model network. However, with skip connection, there exists the essential parameter which doesn't depend on overparametrized layers and the free energy does not also depend on the layer of learning model network.

## 7 Conclusion

In this paper, we studied Free energy of Bayesian Convolutinal Neural Network with Skip Connection and compared to the case without Skip Connection. Free energy of Bayesian CNN with Skip Connection doesn't depend on the layer of the model unlike the case without Skip Connection. In Bayesian learning, the increase of Free energy is equivalent to generalization error, hence the generalization error has same property about the Skip Connection. In particular, Free energy of CNN without skip connection does not depends on the number of parameters in learning network but depends only on that in data generating network. This feature shows the generalization ability of CNN with skip connection does not decrease with respect to any overparameterization in Bayesian learning.