LU-Net: Invertible Neural Networks Based on Matrix Factorization
LU-Net is a simple and fast architecture for invertible neural networks (INN)
that is based on the factorization of quadratic weight matrices
$\mathsf{A=LU}$, where $\mathsf{L}$ is a lower triangular matrix with ones on
the diagonal and $\mathsf{U}$ an upper triangular matrix. Instead of learning a
fully occupied matrix $\mathsf{A}$, we learn $\mathsf{L}$ and $\mathsf{U}$
separately. If combined with an invertible activation function, such layers can
easily be inverted whenever the diagonal entries of $\mathsf{U}$ are different
from zero. Also, the computation of the determinant of the Jacobian matrix of
such layers is cheap. Consequently, the LU architecture allows for cheap
computation of the likelihood via the change of variables formula and can be
trained according to the maximum likelihood principle. In our numerical
experiments, we test the LU-net architecture as generative model on several
academic datasets. We also provide a detailed comparison with conventional
invertible neural networks in terms of performance, training as well as run
time.
Robin Chan, Sarina Penquitt, Hanno Gottschalk
# LU-Net: Invertible Neural Networks Based on Matrix Factorization

###### Abstract

LU-Net is a simple and fast architecture for invertible neural networks (INN) that is based on the factorization of quadratic weight matrices \(A=LU\), where \(L\) is a lower triangular matrix with ones on the diagonal and \(U\) an upper triangular matrix. Instead of learning a fully occupied matrix \(A\), we learn \(L\) and \(U\) separately. If combined with an invertible activation function, such layers can easily be inverted whenever the diagonal entries of \(U\) are different from zero. Also, the computation of the determinant of the Jacobian matrix of such layers is cheap. Consequently, the \(LU\) architecture allows for cheap computation of the likelihood via the change of variables formula and can be trained according to the maximum likelihood principle. In our numerical experiments, we test the \(LU\)-net architecture as generative model on several academic datasets. We also provide a detailed comparison with conventional invertible neural networks in terms of performance, training as well as run time.

 generative models \(\bullet\) invertible neural networks \(\bullet\) normalizing flows \(\bullet\) LU matrix factorization \(\bullet\) LU-Net

## I Introduction

In recent years, learning generative models has emerged as a major field of AI research [1, 2, 3]. In brief, given \(N\) i.i.d. examples \(\{X_{j}\}_{j=1}^{N}\) of a \(\mathbb{R}^{Q}\)-valued random variable \(X\sim P_{X}\), the goal of a generative model is to learn the data distribution \(P_{X}\) while having only access to the sample \(\{X_{j}\}_{j=1}^{N}\). Having knowledge about the underlying distribution of observed data allows for e.g. identifying correlations between observations. Common downstream tasks of generative models then include 1) _density estimation_, i.e. what is the probability of an observation \(x\) under the distribution \(P_{X}\), and 2) _sampling_, i.e. generating novel data points \(x_{new}\) approximately following data distribution \(P_{X}\).

Early approaches include **deep Boltzmann machines**[4, 5], which are energy-based models motivated from statistical mechanics and which learn unnormalized densities. The expressive power of such models is however limited and sampling from the learned distributions becomes intractable even in moderate dimensions due to the computation of a normalization constant, for which the (slow) Markov chain Monte Carlo method is often used instead.

On the contrary, **autoregressive models**[6, 7, 8] learn tractable distributions by relying on the chain rule of probability, which allows for factorizing the density of \(X\) over its \(Q\) dimensions. In this way, the likelihood of data can be evaluated easily, yielding promising results for density estimation and sampling by training via maximum likelihood. Nonetheless, sampling using autoregressive models still remains computationally inefficient given the sequential nature of the inference step over the dimensions.

**Generative adversarial networks** (GANs) avoid the just mentioned drawback by learning a generator in a minimax game against a discriminator [9, 1]. Here, both the generator and discriminator are represented by deep neural networks. In this way, the learning process can be conducted without an explicit formulation of the density of the distribution \(P_{X}\) and therefore can be considered as likelihood-free approach. While GANs have shown successful results on generating novel data points even in high dimensions, the absence of an expression for the density makes them unsuitable for density estimation.

Also **variational autoencoders** (VAEs) have shown fast and successful sampling results [10, 2, 11]. These models consist of an encoder and a decoder part, which are both based on deep neural network architectures. The encoder maps data points \(\{X_{j}\}_{j=1}^{N}\) to lower dimensional latent representations, which define variational distributions. The decoder subsequently maps samples from these variational distributions back to input space. Both parts are trained jointly by maximizing the evidence lower bound (ELBO), i.e. providing a lower bound for the density.

More recently, **diffusion models** have gained increased popularity [12, 13, 14, 15] given their spectacular results. In the forward pass, this type of model gradually adds Gaussian noise to data points. Then, in the backward pass the original input data is recovered from noise using deep neural networks, i.e. representing the sampling direction of diffusion models. However, sampling requires the simulation of a multi stage stochastic process, which is computationally slow compared to simply applying a map. Moreover, just like VAEs, diffusion models do not provide a tractable computation of the likelihood. By training using ELBO, they at least provide a lower bound for the density.

Another type of generative models are **Normalizing flows**[16, 17, 18, 19, 20, 21, 22], which allow for efficient and exact density estimation as well as sampling of high dimensional data. These models learn an invertible transformation
for \(X\sim P_{X}\) such that the distribution \(f^{-1}(Z)\) is as close as possible to the target distribution \(P_{X}\). Here, \(Z\) follows a simple prior distribution \(P_{Z}\) that is easy to evaluate densities with and easy to sample from, such as e.g. multivariate standard normal distribution. Hence, \(f^{-1}\) represents the generator for the unknown (and oftentimes complicated) distribution \(P_{X}\). As normalizing flows are naturally based on the change of variables formula

\[p_{X}(x)=p_{Z}(f(x))\cdot\big{|}\det\left(\Jf(x)\right)\big{|}, \tag{1}\]

with \(X=f^{-1}(Z)\), this expression can also be used for exact probability density evaluation and likelihood based learning.

In this way, normalizing flows offer impressive generative performance along with an explicit and tractable expression of the density. Despite the coupling layers in normalizing flows being rather specific, it has been shown recently that their expressive power is universal for target measures having a density on the target space \(\mathbb{R}^{D}\)[21]. Note however that multiple such coupling layers are usually chained for an expressive generative model. These layers are typically parameterized by the outputs of neural networks and therefore normalizing flows can require considerable computational resources.

In this work, we propose **LU-Net**: an alternative to existing invertible neural network (INN) architectures motivated by the positive properties of normalizing flows. The major advantage of LU-Net is the simplicity of its design. It is based on the elementary insight that a fully connected layer of a feed forward neural network is a bijective map from \(\mathbb{R}^{Q}\to\mathbb{R}^{D}\), if and only if (a) the weight matrix is quadratic, i.e. \(Q=D\), (b) the weight matrix is of full rank and (c) the employed activation function maps \(\mathbb{R}\) to itself bijectively. A straight forward idea is then to fully compose a neural network of such invertible layers and learn a transformation \(f\) as in Eq. (1).

However, it can be computationally expensive to actually invert the just described INN, especially if required during training. Due to the size of weight matrices and the fact that they can be fully occupied, the computational complexity for the inversion scales as \(\mathcal{O}(M\times D^{3})\), where \(M\) is the depth and \(D\) the width of the proposed INN. The same complexity also holds for the computation of the determinant, which occurs in the computation of the likelihood. Both tasks, inversion and computation of the determinant, can be based on the LU-decomposition, where a quadratic matrix is factorized in a lower triangular matrix with ones on the diagonal and an upper triangular matrix with nonzero diagonal entries. Using this factorization inversion becomes of order \(\mathcal{O}(M\times D^{2})\) and computation of the determinant of \(\mathcal{O}(M\times D)\). In the LU-Net architecture, we therefore enforce the weight matrices to be of lower or upper triangular shape and keep this shape fixed during the entire training process. Consequently, a single fully connected layer \(x\mapsto Ax+b\) is decomposed into two layers \(x\mapsto Ux\mapsto UUx+b\), where the weight matrices \(L\) and \(U\) are "masked" on the upper and lower triangular positions, respectively.

This simple architecture of LU-Net however comes with a limitation, that is universal approximation. The classical type of universal approximation theorems deal with a fixed depth and an arbitrary width of neural networks [23, 24, 25]. Obviously, these theorems cannot be applied to LU-Net due its bijectivity constraint. More recent universal approximation theorems deal with a fixed width and an arbitrary depth [26, 27, 28, 29]. But even in the weak sense of \(L^{p}\)-distances, a network requires a width of at least \(D+1\)[29] to be a universal approximator. Thus, the LU-Net just misses the property by one dimension.

One could imagine that the missing dimension in the width of LU-Net becomes less relevant the higher dimensional the problem is, so that the difference between width \(D\) and \(D+1\) becomes marginal. Nevertheless, also for dimensions as low as \(D=2\), we provide numerical evidence that the expressivity of LU-Net still achieves reasonable quality in density estimation.

Moreover, we also present reasonable results using LU-Net for the popular task of generative modeling of images, which includes density estimation and sampling. In a quantitative comparison as suggested by [30], LU-Net achieves a consistent advantage in terms of the negative log likelihood metric when compared to the widely used RealNVP INN architecture [3] with about the same number of parameters. In our experiments we further observe that training LU-Net is computationally considerably cheaper than training the just mentioned coupling layer based normalizing flow. This also points to the particular suitability of LU-Net as base model for rapid prototyping.

Overall, LU-Net provides a simple and efficient framework of an INN. Due to its simplicity, this model is applicable to a variety of problems with data of different forms. This is in contrast to other generative models, e.g. normalizing flows, which are often particularly designed for a specific application such as image generation.

The content of this paper is structured as follows: in Sec. II we describe the LU-Net architecture with details on computing the density and likelihood. Numerical results including experiments on the image datasets MNIST [31] and Fashion-MNIST [32] follow in Sec. III. Finally, in Sec. IV we conclude our article and give recommendations for future research directions.

Additionally, we report evaluations on condition numbers of the LU-layers and on the closeness of the distribution in the normalizing direction of LU-Net to a multivariate standard normal distribution in Appendix A and Appendix B, respectively. The entire code for the numerical experiments presented in this paper is publicly available on GitHub: [https://github.com/spenquitt/LU-Net-Invertible-Neural-Networks](https://github.com/spenquitt/LU-Net-Invertible-Neural-Networks).

## II LU-Net Architecture and the Likelihood

Fully connected neural networks are the most basic models for deep learning, which can be applied to various applications. This generality property is what we also aim for invertible neural networks in the context of probabilistic generative modeling. To this end, we have to ensure that the model is bijective and that the inversion as well as computation of the likelihood are both tractable.
The bijectivity constraint in fully connected neural networks can easily be fulfilled by using a bijective activation function and restricting the weight matrices to be quadratic and of full rank. However, the inversion and computation of the determinant for the computation of the likelihood, cf. 1, of fully occupied matrices remain computationally expensive with a cubic complexity. To address this problem, we propose to directly learn the LU factorization replacing the weight matrices in fully connected layers without a loss in model capacity. This forms the building block of our proposed LU-Net, which we explain in more details in what follows.

### _LU Factorization_

The LU factorization is a common method to decompose any square matrix \(\mathsf{A}\in\mathbb{R}^{\mathsf{D}\times\mathsf{D}}\) into a lower triangular matrix \(\mathsf{L}\in\mathbb{R}^{\mathsf{D}\times\mathsf{D}}\) with ones on the diagonal and in an upper triangular matrix \(\mathsf{U}\in\mathbb{R}^{\mathsf{D}\times\mathsf{D}}\) with non-zero diagonal entries. Then, the matrix \(\mathsf{A}\) can be rewritten as

\[\mathsf{A}=\mathsf{LU}=\left(\begin{array}{ccc}1&&\raisebox{-1.29pt}{ \includegraphics[scale=0.4]{fig-1
The leaky softplus function circumvents those aforementioned limitations, which is why we choose this activation function for all hidden layers in our proposed LU-Net, i.e. \(\phi^{(\mathsf{m})}(\mathsf{x})=\mathsf{LeakySoftplus}(\mathsf{x})\) with \(\alpha=0.1\) for all \(\mathsf{m}=1,\ldots,\mathsf{M}-1\). As we will deal with regression problems, we employ no activation in the final layer, i.e. \(\phi^{(\mathsf{M})}(\mathsf{x})=\mathsf{x}\) is the identity map.

### _Inverse LU-Net_

If all previously described requirements for the inversion of LU-Net are fulfilled, each layer can be reversed as

\[{\mathsf{f}^{(\mathsf{m})}}^{-1}(\mathsf{z})={\mathsf{U}^{(\mathsf{m})}}^{-1}{ \mathsf{L}^{(\mathsf{m})}}^{-1}\left(\phi^{-1}(\mathsf{z})-\mathsf{b}^{( \mathsf{m})}\right)\]

for some input \(\mathsf{z}\in\mathbb{R}^{\mathsf{D}}\) and for all LU layers \(\mathsf{m}=1,\ldots,\mathsf{M}\).

Then, the overall expression for the reversed LU-Net \({\mathsf{f}^{-1}}:\mathbb{R}^{\mathsf{D}}\to\mathbb{R}^{\mathsf{D}}\) is given by

\[{\mathsf{f}^{-1}}(\mathsf{z})=\left({\mathsf{f}^{(1)}}^{-1}\circ\ldots\circ{ \mathsf{f}^{(\mathsf{M}-1)}}^{-1}\circ{\mathsf{f}^{(\mathsf{M})}}^{-1}\right) (\mathsf{z})\]

and represents the "generating direction", see also Fig. 0(b).

Note that both \(\mathsf{f}\) and \({\mathsf{f}^{-1}}\) share their weights and it holds \(\left({\mathsf{f}^{-1}}\circ{\mathsf{f}}\right)(\mathsf{x})=\mathsf{x}\) for any input \(\mathsf{x}\in\mathbb{R}^{\mathsf{D}}\) even without any training.

### _Training via Maximum Likelihood_

Given a dataset \(\mathcal{D}=\{\mathsf{x}^{(\mathsf{n})}\}_{n=1}^{\mathsf{N}}\), containing \(\mathsf{N}\) independently drawn examples of some random variable \(\mathsf{X}\sim\mathsf{P}_{\mathsf{X}}\), our training objective is then to maximize the likelihood

\[\mathscr{L}(\theta|\mathcal{D})=\prod_{n=1}^{\mathsf{N}}\mathsf{p}_{\mathsf{ X}}(\mathsf{x}^{(\mathsf{n})}) \tag{5}\]

where \(\mathsf{p}_{\mathsf{X}}:\mathbb{R}^{\mathsf{D}}\to\mathbb{R}\) denotes the (unknown) probability density function corresponding to the target distribution \(\mathsf{P}_{\mathsf{X}}\) and \(\theta=\{{\mathsf{U}^{(\mathsf{m})}},{\mathsf{L}^{(\mathsf{m})}},{\mathsf{b} ^{(\mathsf{m})}}\}_{n=1}^{\mathsf{M}}\) the set of model parameters of LU-Net. By defining the model function \({\mathsf{f}}(\cdot|\theta)\) of LU-Net to be the invertible transform such that \(\mathsf{X}={\mathsf{f}^{-1}}(\mathsf{Z}|\theta)\Leftrightarrow\mathsf{Z}={ \mathsf{f}}(\mathsf{X}|\theta)\), where \(\mathsf{Z}\sim\mathsf{P}_{\mathsf{Z}}\) is another random variable following a simple prior distribution, we can use the change of variables formula to rewrite the expression in Eq. (5) to

\[\mathscr{L}(\theta|\mathcal{D})=\prod_{n=1}^{\mathsf{N}}\mathsf{p}_{\mathsf{ Z}}\left({\mathsf{f}}(\mathsf{x}^{(\mathsf{n})}|\theta)\right)\big{|}\det \left({\mathbb{J}}{\mathsf{f}}(\mathsf{x}^{(\mathsf{n})}|\theta)\right)\big{|}. \tag{6}\]

As in normalizing flows, we choose \(\mathsf{P}_{\mathsf{Z}}\) to be a \(\mathsf{D}\)-multivariate standard normal distribution with probability density function

\[\mathsf{p}(\mathsf{z})=\frac{1}{\sqrt{(2\pi)^{\mathsf{D}}}}\,\exp\left(- \frac{1}{2}\mathsf{z}^{\intercal}\mathsf{z}\right)\!,\ \mathsf{z}\in\mathbb{R}^{\mathsf{D}}. \tag{7}\]

Further, given the fact that the determinant of a triangular matrix is the product of its diagonal entries and given the chain rule of calculus, for each LU layer \(\mathsf{m}=1,\ldots,\mathsf{M}\) of LU-Net it applies

\[\begin{split}&\big{|}\det\left({\mathbb{J}}{\mathsf{f}^{(\mathsf{ m})}}(\mathsf{x})\right)\big{|}\\ =&\big{|}\prod_{\mathsf{m}=1}^{\mathsf{M}}\prod_{ \mathsf{d}=1}^{\mathsf{D}}\phi^{\prime(\mathsf{m})}\left(({\mathsf{L}}^{( \mathsf{m})}{\mathsf{U}^{(\mathsf{m})}}\mathsf{x})_{\mathsf{d}}+{\mathsf{b} }_{\mathsf{d}}^{(\mathsf{m})}\right)\cdot{\mathsf{u}}_{\mathsf{d},\mathsf{d}}^{ (\mathsf{m})}\big{|}\end{split} \tag{8}\]

with \(\phi^{\prime(\mathsf{m})}\) being the derivative of the \(\mathsf{m}\)-th LU layer's activation function \(\phi^{(\mathsf{m})}\).

Considering now again the chain rule of calculus and taking into account Eq. (6), Eq. (7) as well as Eq. (8), we obtain the following final expression for the negative log likelihood as training loss function:

\[\begin{split}&-\ln\mathscr{L}(\theta|\mathcal{D})\\ =&\frac{1}{2}\cdot\mathsf{N}\cdot\mathsf{D}\cdot\ln(2 \pi)+\frac{1}{2}\sum_{n=1}^{\mathsf{N}}\sum_{\mathsf{d}=1}^{\mathsf{D}} \mathsf{f}_{\mathsf{d}}(\mathsf{x}^{(\mathsf{n})}|\theta)^{2}\\ &-\sum_{n=1}^{\mathsf{N}}\sum_{\mathsf{m}=1}^{\mathsf{M}}\sum_{ \mathsf{d}=1}^{\mathsf{D}}\ln\phi^{\prime(\mathsf{m})}\left(({\mathsf{L}}^{( \mathsf{m})}{\mathsf{U}^{(\mathsf{m})}}\mathsf{x}^{(\mathsf{n})})_{\mathsf{d}}+{ \mathsf{b}}_{\mathsf{d}}^{(\mathsf{m})}\right)\\ &-\mathsf{N}\cdot\sum_{\mathsf{m}=1}^{\mathsf{M}}\sum_{\mathsf{d}=1 }^{\mathsf{D}}\ln\big{|}{\mathsf{u}}_{\mathsf{d},\mathsf{d}}^{(\mathsf{m})} \big{|}\to\min\.\end{split} \tag{9}\]

Note that for each hidden LU layer \(\mathsf{m}=1,\ldots,\mathsf{M}-1\) in LU-Net

\[\phi^{\prime(\mathsf{m})}(\mathsf{x})=\alpha+\frac{1-\alpha}{1+\exp(-\mathsf{x} )}=\alpha+(1-\alpha)\,\sigma(\mathsf{x})\]

with \(\sigma:\mathbb{R}\to\mathbb{R}\) being the logistic sigmoid function. For the final layer the derivative is constant with \(\phi^{\prime(\mathsf{M})}=1\).

## III LU-Net Experiments

In this section we present extensive experiments with LU-Net, which were conducted in different settings. As a toy example, we apply LU-Net to learn a two-dimensional Gaussian mixture. Next, we apply LU-Net to the image datasets MNIST [31] as well as the more challenging Fashion-MNIST [32]. We evaluate LU-Net as density estimator and also have look at the sampling quality as generator.

### _Training goal and evaluation_

In general, our goal is to learn a target distribution \(\mathsf{P}_{\mathsf{X}}\) given only samples which are produced by its data generating process. To this end, we attempt to train our model distribution given by LU-Net to be as close as possible to the data distribution, i.e. the empirical distribution provided by examples \(\mathcal{D}=\{\mathsf{x}^{(\mathsf{n})}\}_{n=1}^{\mathsf{N}}\) of the random variable \(\mathsf{X}\sim\mathsf{P}_{\mathsf{X}}\).

One common way then to quantify the closeness between two distributions \(\mathsf{P}\) and \(\mathsf{Q}\) with probability density functions \(\mathsf{p}\) and \(\mathsf{q}\), respectively, is via the Kullback-Leibler divergence

\[\mathsf{D}_{\text{KL}}(\mathsf{P}||\mathsf{Q})=\mathbb{E}_{\mathsf{X}\sim\mathsf{ P}}\left[\ln\frac{\mathsf{p}(\mathsf{X})}{\mathsf{q}(\mathsf{X})}\right]=\int_{ \mathbb{R}}\mathsf{p}(\mathsf{x})\ln\frac{\mathsf{p}(\mathsf{x})}{\mathsf{q}( \mathsf{x})}\text{d}\text{d}\text{.}\]

It is well known that maximizing the likelihood on the dataset \(\mathcal{D}\), as presented in Eq. (9), asymptotically amounts to minimizing the Kullback-Leibler divergence between the target distribution and model distribution [9, 33], which in our case are defined by \(\mathsf{P}_{\mathsf{X}}\) and LU-Net, respectively. For this reason, the negative log likelihood (NLL) is not only used as loss function for training, but also as the standard metric to measure the density estimation capabilities of probabilistic generative models [30].
### _Gaussian mixture_

#### Iv-B1 Experimental setup

To begin, we create a dataset consisting of 10,000 sampled two-dimensional Gaussian data points at four different centers with a standard deviation of \(0.2\) of which we use 9,000 for training LU-Net in different configurations, see Fig. 2(a). More precisely, we train five LU-Nets in total comprising 2, 3, 5, 8, and 12 hidden LU layers with a final LU output layer each for 10, 20, 30, 35, and 40 epochs, respectively. As optimization algorithm we use stochastic gradient descent with a momentum term of 0.9. We start with a learning rate of \(1.0\) that decays by 0.9 in each training epoch. Further, we clip the gradient to a maximal length of 1 w.r.t. the absolute-value norm, which empirically have shown to stabilize the training process.

#### Iv-B2 Results

In Tab. I we report the negative log likelihood of LU-Net on the 1,000 holdout test data points. In Fig. 2(b) - Fig. 2(f) we provide visualizations of the learned and ground truth density. Generally, we observe that by stacking more LU layers the model function becomes more flexible, which is in line with [29] stating that deeper neural network have increased capacity. In our toy experiments the LU-Net with 12 layers achieves the best result with an averaged NLL of 1.0848. This is visible in the visualization in Fig. 2(f), clearly showing having learned modes in proximity of the true centers of the target Gaussian mixture. We conclude that in practice depth can increase the expressive power of LU-Net.

### _MNIST and Fashion-MNIST_

#### Iv-C1 Data preprocessing

The two publicly available datasets MNIST [31] and Fashion-MNIST [32] consist of gray-scaled images of resolution \(28\times 28\) pixels. These images are stored in 8-bit integers, i.e. each pixel can take on a brightness value from \(\{0,1.\dots,255\}\). Modeling such a discrete data distribution with a continuous model distribution (as we do with LU-Net by choosing a Gaussian prior, cf. Eq. (7)), could lead to arbitrarily high likelihood values, since arbitrarily narrow and high densities could be placed as spikes on each of the discrete brightness values. This practice would make the evaluation via the NLL not comparable and thus meaningless.

Therefore, it is best practice in generative modeling to add real-valued uniform noise \(\mathsf{u}\sim\mathsf{U}(0,1)\), \(\mathsf{u}\in[0,1)\) to each pixel of the images in order to dequantize the discrete data [30, 34]. It turns out that the likelihood of the continuous model on the dequantized data is upper bounded by the likelihood on the original image data [30, 35]. Consequently, maximizing the likelihood on \(\mathsf{x}+\mathsf{u}\) will also maximize the likelihood on the original input \(\mathsf{x}\). This also makes the NLL on the dequantized data a comparable performance measure with NLL \(>0\) for any probabilistic generative model dealing with images. Note that the just described non-deterministic preprocessing step can easily be reverted by simply rounding off.

As additional preprocessing steps we normalize the dequantized pixel values to the unit interval \([0,1]\) by dividing by 256, and apply the logit function to transform the data distribution to a Gaussian-like shape. This output then represents the input to LU-Net. Again, these preprocessing steps can easily be reversed by applying the inverse of logit, i.e. the logistic sigmoid function, and by multiplying by 256, respectively.

#### Iv-C2 Experimental setup

We conduct the experiments on MNIST and Fashion-MNIST with an LU-Net consisting of three hidden LU layers and a final LU output layer. The model is trained for a maximum of 40 epochs and conditioned on each class of MNIST and Fashion-MNIST. As optimization algorithm, we stick to gradient descent with a momentum

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline \hline \multicolumn{3}{|c|}{MNIST Test NLL} & \multicolumn{2}{c|}{Fashion-MNIST Test NLL} \\ \hline Class & Bits \(\mathsf{P}\) Pixel 1 & Class & Bits \(\mathsf{P}\) Pixel 1 \\ \hline Number 0 & 2.7180 \(\pm\) 0.0284 & T-Shirt & 3.7726 \(\pm\) 3.1228 \\ Number 1 & 2.4795 \(\pm\) 0.0125 & Trousers & 7.2577 \(\pm\) 3.49085 \\ Number 2 & 2.9395 \(\pm\) 0.0997 & Pullover & 2.4018 \(\pm\) 0.1091 \\ Number 3 & 2.7465 \(\pm\) 0.0062 & Dress & 2.5337 \(\pm\) 0.0127 \\ Number 4 & 2.7760 \(\pm\) 0.0114 & Coat & 3.4392 \(\pm\) 1.6847 \\ Number 5 & 2.7489 \(\pm\) 0.0175 & Sandal & 2.8861 \(\pm\) 0.0173 \\ Number 6 & 2.8591 \(\pm\) 0.8279 & Shirt & 2.7209 \(\pm\) 0.1560 \\ Number 7 & 2.7930 \(\pm\) 0.0511 & Sneaker & 3.6077 \(\pm\) 0.0071 \\ Number 8 & 2.7916 \(\pm\) 0.0081 & Bag & 4.3983 \(\pm\) 2.0313 \\ Number 9 & 2.6576 \(\pm\) 0.0187 & Ankle Boot & 4.4405 \(\pm\) 0.1762 \\ \hline \hline Average & 2.7480 \(\pm\) 0.2931 & Average & 3.7568 \(\pm\) 2.4632 \\ \hline \end{tabular}
\end{table} TABLE II: Class-wise negative log likelihood (NLL) when applying LU-Net to MNIST and Fashion-MNIST test dataset. The results are averaged over 30 runs. Note that the NLL is reported in bits per pixel (sometimes also called bits per dimension or simply bpd), which is the NLL with logarithm base 2 averaged over all pixels.

Fig. 3: The LU-Net with different numbers of LU layers applied to a two-dimensional Gaussian mixture. The heatmaps indicate the ground truth densities and the level curves the learned densities given by training LU-Net.

\begin{table}
\begin{tabular}{|c|c|c|} \hline \hline \multicolumn{3}{|c|}{Gaussian mixture negative log likelihood} \\ \hline \# LU Layers & 2 & 3 & 5 \\ \hline Test NLL \(\downarrow\) & 3.4024 \(\pm\) 0.3403 & 2.6765 \(\pm\) 0.3607 & 1.8665 \(\pm\) 0.2875 \\ \hline \hline \# LU Layers & 8 & 12 \\ \hline Test NLL \(\downarrow\) & 1.4633 \(\pm\) 0.2132 & 1.0848 \(\pm\) 0.3239 \\ \hline \end{tabular}
\end{table} TABLE I: Test results of LU-Net on the two-dimensional Gaussian mixture toy problem.
parameter of 0.9. We start with a learning rate of 0.6 that decays by 0.5 every three epochs. Further, we clip the gradients to a maximal length of 1 w.r.t. Euclidean norm, which empirically have shown to stabilize the training process as well as regularizing the weights. As extra loss weighting, we add a factor of \(\gamma=100\) to the sum of the diagonal entries in the NLL loss function, see Eq. (9), which has considerably improved the convergence speed of the training process.

#### Iv-B3 Results

In Tab. II we report the negative log likelihood computed by LU-Net on the MNIST and Fashion-MNIST test datasets. In Fig. 4 we provide qualitative examples of LU-Net as density estimator and in Fig. 5 as well as Fig. 6 as generator of new samples.

In terms of numerical results, we achieve adequate density estimation scores with an average NLL of \(2.75\pm 0.29\) bits/pixel and \(3.74\pm 2.46\) bits/pixel over all classes on MNIST and Fashion-MNIST, respectively. In general, the results can be considered as robust for each class of both datasets. The only major deviations are related to the classes T-Shirt and Trousers of Fashion-MNIST, which can be explained by the large variety of patterns in different examples and hence the increased difficulty in generative modeling, cf. also Tab. II right and Fig. 3(b). Furthermore, we notice that LU-Net is capable of assigning meaningful likelihoods, i.e. images that are more characteristic for the associated class are assigned higher likelihoods, see again Fig. 4 in particular.

With regard to LU-Net as generator, we obtain reasonable quality of the sampled images. The random samples can clearly be recognized as subset of MNIST or Fashion-MNIST and moreover, they can also be assigned easily to the corresponding classes. However, we notice that many generated examples contain noise, most visible for class 7 in MNIST or class sandal and bag in Fashion-MNIST, cf. Fig. 4(a) and Fig. 4(b), respectively. This shortcoming is not surprising since the fully connected layers of LU-Net capture less spatial correlations as other filter based architectures commonly used on image data.

Another noteworthy observation refers to the learned latent space of LU-Net, i.e. the space after applying the normalizing sequence. Given its invertibility property, each latent variable represents exactly one image, which allows for traveling though the latent space and thus also interpretation of it. By interpolating between two latent representations, we generally observe a smooth transition between the two corresponding images when transforming back to original input space, see Fig. 5(a) for MNIST and Fig. 5(b) for Fashion-MNIST. This also enables the visual inspection of relevant features or parts of the content associated with certain images or classes.

To conclude, we have seen that LU-Net even with a shallow architecture can be applied as probabilistic generative model to images. The numerical results on these higher dimensional data indicate that the bijectivity constraint is not a significant limitation regarding the expressive power. Although not specifically designed to model image data, LU-Net is still capable of generating sufficiently clear images, which highlights its general purpose property as generative model.

Fig. 4: Unseen original test samples of (a) MNIST and (b) Fashion MNIST. These images are ordered class-wise and in decreasing likelihood from left to right as estimated by LU-Net. See also Tab. II for the class names.

Fig. 5: LU-Net: Randomly generated samples of (a) MNIST and (b) Fashion MNIST. These examples are generated by sampling random numbers from a multivariate normal distribution and passing them through the inverse of LU-Net. Moreover, these images are ordered class-wise and in decreasing likelihood from left to right as estimated by LU-Net.

Fig. 6: LU-Net: Samples of (a) MNIST and (b) Fashion MNIST generated by interpolating in latent space of LU-Net. The examples in the red boxes are reconstructions from latent representations of original test data. Note that the examples are ordered randomly.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline \multicolumn{2}{|c|}{MNIST Test NLL} & \multicolumn{2}{|c|}{Fashion-MNIST Test NLL} \\ \hline \hline Class & Bits / Pixel \(\downarrow\) & Class & Bits / Pixel \(\downarrow\) \\ \hline Number 0 & 5.2647 \(\pm\) 0.0189 & T-Shirt & 6.0803 \(\pm\) 0.0321 \\ Number 1 & 5.0888 \(\pm\) 0.1320 & Turouses & 5.5130 \(\pm\) 0.0778 \\ Number 2 & 6.0978 \(\pm\) 0.0249 & Pullover & 6.1770 \(\pm\) 0.0043 \\ Number 3 & 5.0174 \(\pm\) 1.0864 & Dress & 6.0273 \(\pm\) 0.1005 \\ Number 4 & 5.0952 \(\pm\) 0.0278 & Coat & 6.1355 \(\pm\) 0.0515 \\ Number 5 & 5.4285 \(\pm\) 0.0742 & Sandal & 5.9324 \(\pm\) 0.0449 \\ Number 6 & 5.4642 \(\pm\) 0.0393 & Shirt & 6.2517 \(\pm\) 0.0437 \\ Number 7 & 5.7739 \(\pm\) 0.0040 & Sneaker & 5.6603 \(\pm\) 0.0007 \\ Number 8 & 5.4086 \(\pm\) 0.0555 & Bag & 6.3035 \(\pm\) 0.0478 \\ Number 9 & 5.0607 \(\pm\) 0.0012 & Ankle Boot & 5.8741 \(\pm\) 0.0314 \\ \hline \hline Average & 5.3682 \(\pm\) 0.1464 & Average & 5.59956 \(\pm\) 0.0386 \\ \hline \end{tabular}
\end{table} TABLE III: Class-wise negative log likelihood in bits per pixel when applying RealNVP to MNIST and Fashion-MNIST test dataset. The results are averaged over 30 runs.
### _Comparison with RealNVP_

In these final experiments we want to compare LU-Net with the popular and widely used normalizing flow architecture RealNVP [3]. To make the models better comparable, we design them to be of similar size in terms of model parameters. In more detail, we employ a RealNVP normalizing flow with 9 affine coupling layers with checkerboard mask mixing. For the coupling layers two small ResNets [36] are used to compute the scale and translation parameter, respectively. Here, every ResNet backbone consists of two residual blocks, each applying two \(3\times 3\) convolutions with 64 kernels and ReLU activations. In total this amounts to a normalizing flow with 5,388,264 weight parameters compared to 4,920,384 weight parameters with LU-Net. Finally, we conducted the same experiments as presented in Sec. III-C for RealNVP.

In Tab. III we report the negative log likelihood scores of our implemented RealNVP on the test splits of MNIST as well as Fashion-MNIST. In comparison to LU-Net, we observe more robust results but overall significantly worse performance in density estimation for each class of both datasets with RealNVP. During the experiments with RealNVP, we realized that the model needs to treated carefully as slight modifications in the hyper parameters could quickly lead to unstable training. We ended up training the flows for 100 epoch using a small learning rate of 1e-4 that decays by 0.2 every 10 epochs.

Besides the worse performance on learning the data distribution of MNIST and Fashion-MNIST, RealNVP further is computationally more expensive than LU-Net, which can be seen in Tab. IV showing a comparison of the computational budget. RealNVP not only requires more GPU memory than LU-Net but also considerably more time to train. The latter point can be explained by backpropagation not working as efficiently due to the deep neural networks employed in coupling layers, which adversely affects propagating the errors from layer to layer. Moreover, RealNVP is notably slower at density evaluation, with LU-Net being 7 times faster. With regard to sampling, RealNVP is however significantly faster by nearly 50 times. Here, we want to note that at run time linear equation systems are solved in inverse LU-Net instead of inverting the weight matrices, cf. Sec. II-D and Appendix C, respectively. Although the inversion could be performed offline saving a considerable amount of operations, the computation of big inverse matrices is often numerically unstable, which is highly undesirable in particular in the context of invertible neural networks and therefore omitted.

Lastly, we present qualitative examples of RealNVP as density estimator in Fig. 7 and as generator in Fig. 8 for MNIST and Fashion-MNIST. At first glance, it is directly notable that the generated images are less noisy in comparison to the images generated by LU-Net. This might be an consequence of the extensive use of convolution operations in RealNVP that helps the model to better capture local correlations of features in images. However, the shapes of the digits and clothing articles in the images generated by our implemented slim RealNVP are still rather unnatural, which can be solved by deeper RealNVP models [3].

## IV Conclusion and Outlook

We introduced LU-Net, which is a simple architecture for an invertible neural network based on the LU-factorization of weight matrices and invertible as well as two times differentiable activation functions. LU-Net provides an explicit likelihood evaluation and reasonable sampling quality. The execution of both tasks is computationally cheap and fast, which we tested in several experiments on academic data sets.

In the future, we intend to investigate more closely the effects of the choice of the activation function. LU-net would become even simpler if leaky ReLu activation functions could be used, which can be achieved by training adversarially. Also, we intend to revisit the universal approximation properties for LU-Nets using artificial widening via zero padding.

Fig. 8: RealNVP: Randomly generated samples of (a) MNIST and (b) Fashion MNIST. These images are ordered class-wise and in decreasing estimated likelihood from left to right.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline  & num weight & GPU memory & num epochs & test NLL \\ model & parameters & usage in MiB & training & in bit/pixel \\ \hline \hline LU-Net & 4.92M & 1,127 & 40 & 3,242 \\ RealNVP & 5.39M & 3,725 & 100 & 5,6819 \\ \hline \hline  & train epoch & optimization & density per & sampling per \\ model & in sec & step in ms & image in ms & image in ms \\ \hline \hline LU-Net & 7.32 & 1.2 & 37.10 & 45.15 \\ RealNVP & 99.88 & 56.0 & 259.15 & 1.03 \\ \hline \end{tabular}
\end{table} TABLE IV: Model size and run time comparisons between LU-Net and RealNVP. Note that the time results are averages over 100 runs on MNIST and Fashion-MNIST. Moreover, all these tests were conducted on the same machine using an NVIDIA Quadro RTX 8000 GPU and a batch size of 128.

Fig. 7: Unseen original test samples of (a) MNIST and (b) Fashion MNIST. These images are ordered class-wise and in decreasing likelihood from left to right as estimated by RealNVP. Note that RealNVP incorrectly assigns high likelihoods to some unclear examples, and vice versa.
## Acknowledgment

This work has been funded by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) via the research consortium AI Delta Learning (grant no. 19A19013Q) and the Ministry of Culture and Science of the German state of North Rhine-Westphalia as part of the KI-Starter research funding program (grant no. 005-2204-0023). Moreover, this work has been supported by the German Federal Ministry of Education and Research (grant no. 01IS22069).
