Continuous Indeterminate Probability Neural Network
This paper introduces a general model called CIPNN - Continuous Indeterminate
Probability Neural Network, and this model is based on IPNN, which is used for
discrete latent random variables. Currently, posterior of continuous latent
variables is regarded as intractable, with the new theory proposed by IPNN this
problem can be solved. Our contributions are Four-fold. First, we derive the
analytical solution of the posterior calculation of continuous latent random
variables and propose a general classification model (CIPNN). Second, we
propose a general auto-encoder called CIPAE - Continuous Indeterminate
Probability Auto-Encoder, the decoder part is not a neural network and uses a
fully probabilistic inference model for the first time. Third, we propose a new
method to visualize the latent random variables, we use one of N dimensional
latent variables as a decoder to reconstruct the input image, which can work
even for classification tasks, in this way, we can see what each latent
variable has learned. Fourth, IPNN has shown great classification capability,
CIPNN has pushed this classification capability to infinity. Theoretical
advantages are reflected in experimental results.
Tao Yang
# Continuous Indeterminate Probability Neural Network

###### Abstract

This paper introduces a general model called **CIPNN** - **C**ontinuous **I**ndeterminate **P**robability **N**eural **N**etwork, and this model is based on IPNN, which is used for discrete latent random variables. Currently, posterior of continuous latent variables is regarded as intractable, with the new theory proposed by IPNN this problem can be solved. Our contributions are Four-fold. First, we derive the analytical solution of the posterior calculation of continuous latent random variables and propose a general classification model (CIPNN). Second, we propose a general auto-encoder called CIPAE - Continuous Indeterminate Probability Auto-Encoder, the decoder part is not a neural network and uses a fully probabilistic inference model for the first time. Third, we propose a new method to visualize the latent random variables, we use one of N dimensional latent variables as a decoder to reconstruct the input image, which can work even for classification tasks, in this way, we can see what each latent variable has learned. Fourth, IPNN has shown great classification capability, CIPNN has pushed this classification capability to infinity. Theoretical advantages are reflected in experimental results. (Source code: [https://github.com/Starfruit007/cipnn](https://github.com/Starfruit007/cipnn))

Although recent breakthroughs demonstrate that neural networks are remarkably adept at natural language processing [28, 11, 21], image processing [14], neural networks are still black-box for human [6], cognitive scientists and neuroscientist have argued that neural networks are limited in their ability to represent variables and data structures [12, 4]. Probabilistic models are mathematical descriptions of various natural and artificial phenomena learned from data, they are useful for understanding such phenomena, for prediction of unknowns in the future, and for various forms of assisted or automated decision making [20].

Deep Latent Variable Models (DLVMs) is a probabilistic model and can refer to the use of neural networks to perform latent variable inference [18]. Currently, the posterior calculation is regarded as intractable [19, 20], and the variational inference method is used for efficient approximate posterior inference [19, 27, 23].

IPNN - Indeterminate Probability Neural Network [31] proposed a new theory, which is used to derive the analytical solution of the posterior calculation of discrete random variables. However, IPNN need predefine the sample space of each discrete random variable (called'split shape' in IPNN), it is sometimes hard to define a proper sample space for an unknown dataset. For CIPNN, the sample space of each continuous random variable is infinite, this issue will not exit in CIPNN.

The rest of this paper is organized as follows: In Sec. 1, related work of VAE and IPNN is introduced. In Sec. 2, the analytical solution of CIPNN is derived and the regularization method is discussed. In Sec. 3, CIPAE is derived and we propose a new method to visualize each latent variable. In Sec. 4, we discuss the training strategy, and two common training setups are discussed: CIPNN and CIPAE are combined together for better evaluation of classification and auto-encoder tasks. In Sec. 5, CIPNN and CIPAE are evaluated and the latent variables are visualized with our new proposed method. Finally, we put forward some future research ideas and conclude the paper in Sec. 6.

## 1 Related Work

### Vae

Modern machine learning and statistical applications require large scale inference in complex models, the inference model are regarded as intractable and either Markov Chain Monte Carlo (MCMC) [24] or variational Bayesian inference [17] are used as approximate solutions [27]. VAE [19] proposes an estimator of the variational lower bound for efficient approximate inference with continuous latent variables. DARN method is generative auto-encoder capable of learning hierarchies of distributed representations from data, and their method applies to binary latent variables [13]. In concurrent work of VAE, two later independent papers proposed equivalent algorithms [27, 23], which
provides an additional perspective on VAE and the latter work applies also the same reparameterization method. Two methods proposed by VAE are also used to realize our analytical solution: the reparameterization trick for making the model differentiable and the KL divergence term for regularization.

VAEs have been used for many tasks such as image generation [22], anomaly detection [30] and de-noising tasks [16][5]. The drawback of auto-encoder is its strong tendency to over-fit [26], as it is solely trained to encode and decode with as little loss as possible regardless of how the latent space is organized [32], VAE has been developed as an effective solutions [26, 2], e.g. VAEs has been used in EEG classification tasks to learn robust features [33, 1, 2, 3].

The framework of our CIPAE is almost the same as that of VAE, the only difference is that VAE uses neural network as the approximate solution of decoder, while CIPAE uses probabilistic model as the analytical solution of decoder.

### Ipin

Let \(X\in\left\{x_{1},x_{2},\ldots,x_{n}\right\}\) be training samples (\(x_{k}\) is understood as ID of random experiment - select one train sample) and \(Y\in\left\{y_{1},y_{2},\ldots,y_{m}\right\}\) consists of m discrete labels (or classes), \(P(y_{l}|x_{k})=y_{l}(k)\) describes the label of sample \(x_{k}\). For prediction, the posterior of the label for a given new input sample \(x_{t}\) is formulated as \(P^{Z}\left(y_{l}\mid x_{t}\right)\), superscript \(Z\) stands for the medium - model outputted N-dimensional random variables \(Z=\left(z^{1},z^{2},\ldots,z^{N}\right)\), via which we can infer label \(y_{l},l=1,2,\ldots,m\).

The analytical solution of the posterior is as bellow [31]:

\[P^{Z}\left(y_{l}\mid x_{t}\right)=\int\limits_{Z}\left(P\left(y_{l}\mid Z \right)\cdot\prod\limits_{i=1}^{N}P\left(z^{i}\mid x_{t}\right)\right) \tag{1}\]

Where,

\[P\left(y_{l}\mid Z\right)=\frac{\sum_{k=1}^{n}\left(P\left(y_{l}\mid x_{k} \right)\cdot\prod_{i=1}^{N}P\left(z^{i}\mid x_{k}\right)\right)}{\sum_{k=1}^{ n}\prod_{i=1}^{N}P\left(z^{i}\mid x_{k}\right)} \tag{2}\]

## 2 CIPNN

### Continuous Indeterminate Probability

Figure 1 shows CIPNN model architecture, the neural network is used to output the parameter \(\theta\) of some prior distribution of continuous random variable \(z^{i},i=1,2,\ldots,N\). All the random variables together form the N-dimensional joint sample space, marked as \(Z=\left(z^{1},z^{2},\ldots,z^{N}\right)\), and the whole joint sample space are fully connected with all labels \(Y\in\left\{y_{1},y_{2},\ldots,y_{m}\right\}\) via conditional probability \(P\left(y_{l}\mid z^{1},z^{2},\ldots,z^{N}\right)\).

For each continuous random variable \(z^{i}\), the indeterminate probability is formulated as:

\[P\left(z^{i}\mid x_{k}\right)=p\left(z;\theta_{k}^{i}\right),i=1,2,\ldots,N. \tag{3}\]

Where \(z\) is generated from some prior distribution with parameter \(\theta_{k}^{i}\), and \(p\left(z;\theta_{k}^{i}\right)\) is also the density function of \(P\left(z^{i}\mid x_{k}\right)\).

Substitute \(P(y_{l}|x_{k})=y_{l}(k)\) and Eq. (3) into Eq. (1):

\[\begin{split}& P^{Z}\left(y_{l}\mid x_{t}\right)\\ &=\int\limits_{Z}\left(\frac{\sum_{k=1}^{n}\left(y_{l}(k)\prod_{i }^{N}p\left(z;\theta_{k}^{i}\right)\right)}{\sum_{k=1}^{n}\left(\prod_{i}^{N}p \left(z;\theta_{k}^{i}\right)\right)}\prod_{i}^{N}p\left(z;\theta_{t}^{i} \right)\right)\\ &=\mathbb{E}_{z^{\sim}p\left(z;\theta_{t}^{i}\right)}\left(\frac {\sum_{k=1}^{n}\left(y_{l}(k)\cdot\prod_{i}^{N}p\left(z;\theta_{k}^{i}\right) \right)}{\sum_{k=1}^{n}\left(\prod_{i}^{N}p\left(z;\theta_{k}^{i}\right) \right)}\right)\end{split} \tag{4}\]

As the integration over \(Z\) is complicated, \(P^{Z}\left(y_{l}\mid x_{t}\right)\) is rewritten as expectation, we can then use Monte Carlo method [24] to make an approximate estimation. However, a directly sampling \(z\) from distribution \(p\left(z;\theta_{t}^{i}\right)\) will make the exception not differentiable. Hence, we use the reparameterization trick [19]: let \(\varepsilon\sim p\left(\varepsilon\right)\) be some random noise, and define a mapping function \(z=g(\varepsilon,\theta)\), so \(p\left(z;\theta_{k}^{i}\right)\) can be rewritten as \(p\left(g(\varepsilon,\theta);\theta_{k}^{i}\right)\).

Therefore, together with Monte Carlo method, the above function can be further formulated as:

Figure 1: CIPNN – model architecture. Where \(P\left(y_{l}\mid z^{1},z^{2},\ldots,z^{N}\right)\) is statistically calculated, not model weights.
\[P^{Z}\left(y_{l}\mid x_{t}\right) \tag{5}\] \[=\mathbb{E}_{\varepsilon\sim p\left(\varepsilon\right)}\left(\frac{ \sum_{k=1}^{n}\left(y_{l}(k)\prod_{i}^{N}p\left(g(\varepsilon,\theta_{t}^{i}); \theta_{k}^{i}\right)\right)}{\sum_{k=1}^{n}\left(\prod_{i}^{N}p\left(g( \varepsilon,\theta_{t}^{i});\theta_{k}^{i}\right)\right)}\right)\] \[\approx\frac{1}{C}\sum_{c=1}^{C}\left(\frac{\sum_{k=1}^{n}\left(y_ {l}(k)\cdot\prod_{i}^{N}p\left(g(\varepsilon_{c},\theta_{t}^{i});\theta_{k}^{i }\right)\right)}{\sum_{k=1}^{n}\left(\prod_{i}^{N}p\left(g(\varepsilon_{c}, \theta_{t}^{i});\theta_{k}^{i}\right)\right)}\right)\]

Where \(\varepsilon_{c}\sim p\left(\varepsilon\right)\).

Take, for example, the univariate Gaussian case: let \(P\left(z^{i}\mid x_{k}\right)=\mathcal{N}\left(z;\mu_{k}^{i},\sigma_{k}^{i}\right)\), and \(\varepsilon_{c}\sim\mathcal{N}\left(0,1\right)\), the reparameterization mapping function is \(z=\mu+\sigma\varepsilon\), the above function can be rewritten as:

\[P^{Z}\left(y_{l}\mid x_{t}\right) \tag{6}\] \[\approx\frac{1}{C}\sum_{c=1}^{C}\left(\frac{\sum_{k=1}^{n}\left(y _{l}(k)\cdot\prod_{i}^{N}\mathcal{N}\left(g\left(\varepsilon_{c},\theta_{t}^ {i}\right);\theta_{k}^{i}\right)\right)}{\sum_{k=1}^{n}\left(\prod_{i}^{N} \mathcal{N}\left(g\left(\varepsilon_{c},\theta_{t}^{i}\right);\theta_{k}^{i} \right)\right)}\right)\]

Where \(\theta_{k}^{i}:=\left(\mu_{k}^{i},\sigma_{k}^{i}\right)\), \(g\left(\varepsilon_{c},\theta_{t}^{i}\right)=\mu_{t}^{i}+\sigma_{t}^{i}\cdot \varepsilon_{c}\) and \(\varepsilon_{c}\sim\mathcal{N}\left(0,1\right)\).

We use cross entropy as main loss function:

\[\mathcal{L}_{1}=-\sum_{l=1}^{m}\left(y_{l}(t)\cdot\log P^{Z}\left(y_{l}\mid x _{t}\right)\right) \tag{7}\]

### Regularization

The sufficient and necessary condition of achieving global minimum is already proved in IPNN [31], which is also valid for continuous latent variables:

**Proposition 1**: _For \(P(y_{l}|x_{k})=y_{l}(k)\in\left\{0,1\right\}\) hard label case, CIPNN converges to global minimum only when \(P\left(y_{l}|z^{1},z^{2},\ldots,z^{N}\right)\to 1,\) for \(\prod_{i}^{N}p\left(z;\theta_{k}^{i}\right)>0\)._

_In other word, each N-dimensional joint sample area (collection of adjacent joint sample points) corresponds to an unique category. However, a category can correspond to one or more joint sample areas._

According to above proposition, the reduction of training loss will minimize the overlap between conditional joint distribution \(\prod_{i}^{N}p\left(z;\theta_{k}^{i}\right)\) of each category. For Gaussian distribution, the variance will be close to zero, and the conditional joint distribution of each category will be far away from each other. This will cause over-fitting problem [32, 26], we have follow up assumption to avoid over-fitting problem:

**Assumption 1**: _For Gaussian distribution, if the distance between the centers of any two adjacent categories' conditional joint distribution \(\prod_{i}^{N}\mathcal{N}\left(z;\mu_{k}^{i},\sigma_{k}^{i}\right)\) is equal to e.g. \(6\sigma\), then the over-fitting problem can be avoided._

_In this way, the \(6\sigma\) distance will lead to a very small overlap between each category, but the conditional joint distribution of each category will be closely connected with each other. Although the N-dimensional joint sample space is infinite, the effective conditional joint distributions are in a very small space._

VAE uses an additional regularization loss to avoid the over-fitting problem [19, 20], and there are follow up works which has proposed to strengthen this regularization term, such as \(\beta\)-VAE [15, 7], \(\beta\)-TCVAE [8], etc. In order to realize the above assumption, we have a modification of VAE regularization loss:

\[\mathcal{L}_{2}=\sum_{i=1}^{N}\left(D_{KL}\left(\mathcal{N}\left(z;\mu_{k}^{i },\sigma_{k}^{i}\right)||\mathcal{N}\left(z;\gamma\cdot\mu_{k}^{i},1\right) \right)\right) \tag{8}\] \[=\frac{1}{2}\sum_{i=1}^{N}\left(((1-\gamma)\cdot\mu_{k}^{i})^{2}+ (\sigma_{k}^{i})^{2}-\log((\sigma_{k}^{i})^{2})-1\right)\]

Where \(N\) is the dimensionality of \(Z\), regularization factor \(\gamma\in[0,1]\) is a hyperparameter and is used to constrain the conditional joint distribution of each category to be closely connected with each other, impact analysis of regularization factor \(\gamma\) see Figure 8.

The overall loss is:

\[\mathcal{L}=\mathcal{L}_{1}+\mathcal{L}_{2} \tag{9}\]

## 3 Cipae

For image auto-encoder task, we firstly transform the pixel value to \([0,1]\) (Bernoulli distribution), and let \(Y^{j}\in\{y_{1}^{j},y_{2}^{j}\}_{j=1}^{J}\), where \(J\) is the number of all pixels of one image. \(P(y_{1}^{j}|x_{k})=p_{1}^{j}(k)\in[0,1]\), which describes the pixel value of image \(x_{k}\) at \(j^{th}\) position, and \(P(y_{2}^{j}|x_{k})=p_{2}^{j}(k)=1-p_{1}^{j}(k)\).

Substitute \(P(y_{l}^{j}|x_{k})\) into Eq. (6), we will get \(P^{Z}\left(y_{l}^{j}\mid x_{t}\right),l=1,2\). In this way, the reconstructed image is formulated as:

\[\text{reconstructed image}:=\left\{P^{Z}\left(y_{1}^{j}\mid x_{t}\right) \right\}_{j=1}^{J} \tag{10}\]

In addition, with one (or part) of N dimensional latent variables we can also reconstruct the input image, the reconstructed image is:1

Footnote 1: The details of applying the superscript \(z^{i}\) are discussed in IPNN [31].
\[\text{reconstructed feature}:=\left\{P^{z^{i}}\left(y_{1}^{j}\mid x_{t}\right) \right\}_{j=1}^{J} \tag{11}\]

Where \(i=1,2,\ldots,N\). In this way, we can see what each latent variable has learned.

Substitute Eq. (10) into Eq. (7), we can get a binary cross entropy loss:

\[\mathcal{L}_{1}=-\frac{1}{J}\sum_{j=1}^{J}\sum_{l=1}^{2}\left(p_{l}^{j}(t) \cdot\log P^{Z}\left(y_{l}^{j}\mid x_{t}\right)\right) \tag{12}\]

And substitute the above loss into Eq. (9), we get the overall loss for auto-encoder training.

## 4 Training

In this section, we will focus on the training strategy of gaussian distribution.

### Training Strategy

Given an input sample \(x_{t}\) from a mini batch, with a minor modification of Eq. (6):

\[P^{Z}\left(y_{l}\mid x_{t}\right)\approx\frac{1}{C}\sum_{c=1}^{C}\left(\frac{ \max(H(\varepsilon_{c}),\epsilon)}{\max(G(\varepsilon_{c}),\epsilon)}\right) \tag{13}\]

Where stable number \(\epsilon\) on the denominator is to avoid dividing zero, \(\epsilon\) on the numerator is to have an initial value of 1. Besides,

\[H(\varepsilon_{c}) =\sum_{k=t_{0}}^{t_{1}}\left(y_{l}(k)\prod_{i}^{N}\mathcal{N} \left(\mu_{t}^{i}+\sigma_{t}^{i}\cdot\varepsilon_{c};\mu_{k}^{i},\sigma_{k}^ {i}\right)\right) \tag{14}\] \[G(\varepsilon_{c}) =\sum_{k=t_{0}}^{t_{1}}\left(\prod_{i}^{N}\mathcal{N}\left(\mu_ {t}^{i}+\sigma_{t}^{i}\cdot\varepsilon_{c};\mu_{k}^{i},\sigma_{k}^{i}\right)\right) \tag{15}\]

Where \(t_{0}=\max(0,t_{1}-T)\), \(t_{1}\) is the number of input samples, \(\varepsilon_{c}\sim\mathcal{N}\left(0,1\right)\). Hyperparameter T is for forgetting use, i.e., \(P^{Z}\left(y_{l}\mid x_{t}\right)\) are calculated from the recent T samples. The detailed algorithm implementation is shown in Algorithm (1).

```
0: A sample \(x_{t}\) from mini-batch
0: Parameter: Latent variables dimension \(N\), forget number \(T\), Monte Carlo number \(C\), regularization factor \(\gamma\), stable number \(\epsilon\), learning rate \(\eta\).
0:\(P^{Z}\left(y_{l}\mid x_{t}\right)\)
1: Declare \(\Theta\) as a recorder.
2:for\(k=1,2,\ldots\) Until Convergence do
3: Use \(\Theta\) to record: \(y_{l},\mu_{k}^{i},\sigma_{k}^{i},i=1,2,\ldots,N\).
4:if\(len(\Theta)>T\)then
5: Forget: Reserve recent T elements from \(\Theta\)
6:endif
7: Compute posterior with Eq. (13): \(P^{Z}\left(y_{l}\mid x_{t}\right)\)
8: Compute loss with Eq. (9): \(\mathcal{L}(W)\)
9: Update model parameter: \(W=W-\eta\nabla\mathcal{L}(W)\)
10:endfor
11:return model and the posterior
```

**Algorithm 1** CIPNN or CIPAE training

### Training Setups

By comparing CIPNN and CIPAE, we can see that they can share the same neural network for a training task. As shown in Figure 2, the latent variables of a classification task can be visualized with CIPAE, and we can also use CIPNN to evaluate the performance of an auto-encoder task.

## 5 Experiments and Results

To evaluate the effectiveness of the proposed approach, we conducted experiments on MNIST [10], Fashion-MNIST [29] and Dogs-vs.-Cats-Redux [9] datasets.

Besides, VAE validated that Monte Carlo number \(C\) can be set to 1 as long as the batch size is high enough (e.g. 100) [19], we will set batch size to 64, Monte Carlo number \(C=2\) and forget number \(T=3000\) for all the experiments in this paper.

### Results of Classification Tasks

In this section, we use train setup in Figure 1(a) to perform different classification tasks in order to reconstruct the latent variable to see what they have learned.

The results from the work [31] show that IPNN prefers to put number 1,4,7,9 into one cluster and the rest into another cluster. We also get a similar interesting results in

Figure 2: Training setups for classification and auto-encoder tasks. (a) CIPNN is used to do supervised classification tasks and CIPAE is used to reconstruct the input image to see what each latent variable has learned. (b) CIPAE is used to do auto-encoder task and CIPNN is used to evaluate its performance.
CIPNN, as shown in Figure 3, with stable number \(\epsilon=1\), the reconstructed image with 1-D latent space shows a strong tendency to sort the categories into a certain order and the number 1,4,7,9 stays together in the latent space. Similar results are also found with 2-D latent space, see Figure 5. Unfortunately, we currently do not know how to evaluate this sort tendency numerically.

With the visualization method proposed in Eq. (11), we can see what each latent variable has learned in the 10-D latent space. As shown in Figure 4, each latent variable focuses on mainly one or two different categories.

As shown in Figure 5(a,d,g), with a proper regularization factor \(\gamma\), the test dataset is mapped to a relative small latent space, and the over-fitting problem is avoided. Besides, in Figure 5(b,e,h) each joint sample area correspond to one unique category, this is consistent with our Proposition 1. In Figure 5(c,f,i), the reconstructed image follows the conditional joint distribution \(P\left(y_{l}\mid z^{1},z^{2}\right),l=0,2,\ldots,9\).

### Results of Auto-Encoder Tasks

In this section, we will make a comparison with our CIPAE and VAE [19] model using train setup in Figure 2b, for VAE model, we replace CIPAE part with VAE, in order to be able evaluate it with CIPNN model. Besides, the regularization loss of VAE is switched to our proposed loss, see Eq. (8). As shown in Figure 6, the results of auto-encoder tasks between CIPAE and VAE are similar, this result further verifies that CIPAE is the analytical solution.

## 6 Conclusion

General neural networks, such as FCN, Resnet [14], Transformer [28], can be understood as a complex mapping function \(f:X\to Y\)[25], but they are black-box for human [6]. Our proposed model can be understood as two part: \(f:X\to Z\) and \(P(Y\mid Z):Z\to Y\), the first part is still black box for us, but the latter part is not unknown anymore. Such kind of framework may have two advantages: the first part can be used to detect the attributes of datasets and summarize the common part of different categories, as shown in Figure 3; the latter part is a probabilistic model, which may be used to build a large Bayesian network for complex reasoning tasks.

Besides, our proposed framework is quite flexible, e.g. from \(X\) to \(Z\), we can use multiple neural networks with different structures to extract specific attributes as different random variables \(z^{i}\), and these random variables will be combined in the statistical phase.

Although our proposed model is derived from indeterminate probability theory, we can see Determinate from the expectation form in Eq. (4). Finally, we'd like to finish our paper with one sentence: The world is determined with all Indeterminate.

## Acknowledgment

I'd like to thank Chuang Liu, Xiaofeng Ma, Weijia Lu, Ning Wu, Bingyang Li, Zhifei Yang, Peng Liu, Lin Sun, Xiaodong Zhang and Can Zhang for their review of this paper.

## Appendix A Visualization

Figure 7 shows classification results of 20-D latent space on Dogs-vs.-Cats-Redux dataset, we can see that each latent variable focuses on both two different categories.

Impact analysis of regularization factor \(\gamma\) is discussed in Figure 8.
