Efficient Finite Initialization for Tensorized Neural Networks
We present a novel method for initializing layers of tensorized neural
networks in a way that avoids the explosion of the parameters of the matrix it
emulates. The method is intended for layers with a high number of nodes in
which there is a connection to the input or output of all or most of the nodes,
we cannot or do not want to store/calculate all the elements of the represented
layer and they follow a smooth distribution. This method is equally applicable
to normalize general tensor networks in which we want to avoid overflows.
  The core of this method is the use of the Frobenius norm and the partial
lineal entrywise norm of reduced forms of the layer in an iterative partial
form, so that it has to be finite and within a certain range. These norms are
efficient to compute, fully or partially for most cases of interest. In
addition, the method benefits from the reuse of intermediate calculations. We
apply the method to different layers and check its performance. We create a
Python function to run it on an arbitrary layer, available in a Jupyter
Notebook in the i3BQuantum repository:
https://github.com/i3BQuantumTeam/Q4Real/blob/e07c827651ef16bcf74590ab965ea3985143f891/Quantum-Inspired%20Variational%20Methods/TN_Normalizer.ipynb
Alejandro Mata Ali, Iñigo Perez Delgado, Marina Ristol Roura, Aitor Moreno Fdez. de Leceta
# Efficient Finite Initialization for Tensorized Neural Networks

###### Abstract

We present a novel method for initializing layers of tensorized neural networks in a way that avoids the explosion of the parameters of the matrix it emulates. The method is intended for layers with a high number of nodes in which there is a connection to the input or output of all or most of the nodes.

The core of this method is the use of the Frobenius norm of this layer in an iterative partial form, so that it has to be finite and within a certain range. This norm is efficient to compute, fully or partially for most cases of interest. We apply the method to different layers and check its performance. We create a Python function to run it on an arbitrary layer, available in a Jupyter Notebook in the i3BQ quantum repository github.com/i3BQantumTeam/Q4Real

## 1 Introduction

Deep neural networks in the world of machine learning are widely used to obtain good results for use cases in industry, research and various fields. However, for highly complex cases we have the problem of needing a large number of parameters, with very large layers of neurons, which can be seen as having to apply very large matrices. In the literature it has been extensively studied to reduce the number of parameters in various ways, such as decomposing matrices into tensor networks [1] or directly training the tensor network itself [2][3][4] (Fig. 1).

Our focus of analysis will be on methods where a tensor network is generated to model the layer tensor and trained directly, rather than using a full matrix. For example, when we try to use tensorized physics informed neural networks to solve differential equations for big industrial cases, as the heat equation of an engine or fluids in a turbine. In this case the initialization problem is often encountered, which we will see in the next section. If we initialize the elements of each tensor with a certain distribution, when we contract the tensor network to obtain the tensor it represents, some of its elements are too large (infinite) or too small (null) for the computer.

We want to eliminate precisely these problems. A first proposal could be to contract the tensor network and eliminate these elements. However, in certain very large layers we cannot store all the tensor elements in memory, so we need another way.

One way is to re-initialize the tensor network by changing a distribution with better hyperparameters, changing the mean and standard deviation. Nevertheless, many of these methodologies are not easy to apply or are not efficient at all.

Our method consists of iteratively calculating the Frobenius norm for different sections of the tensor network until a condition is met, when we

Figure 1: Arbitrary tensor network layer.
divide all the parameters of the tensor network by the calculated factor in a particular way. This allows us to gradually make the Frobenius norm of the layer tend to the number we want, without having to repeatedly re-initialize.

This method is remarkably interesting for hierarchical tree form layers, especially in Tensor Train (TT), Tensor Train Matrix (TT-M) and Projected Entangled Pair States (PEPS). This can also be used in other methods with tensor networks, such as combinatorial optimization, to determine hyperparameters, and it can be combined with other initialization methods.

## 2 Description of the problem

When we have a tensor network of \(N\) nodes, we will have that the elements of the tensor representing the tensor network are given by the sum of a set of values, each given by the product of \(N\) elements of the different nodes. If we look at the case of a TT layer, as in Fig. 2.a, the shape of the elements of the layer is given as

\[T_{ijklm}=\sum_{nopq}T_{in}^{0}T_{njo}^{1}T_{ojp}^{2}T_{pjq}^{3}T_{qm}^{4}. \tag{1}\]

We see that for 5 indices in the tensor we have to multiply 5 tensor elements, but in the general case with \(N\) indices we have

\[T_{ioi_{1}\ldots i_{N-1}}=\sum_{j_{0}j_{1}\cdots j_{N-2}}T_{i_{0}j_{0}}^{0}T_{ j_{0}i_{1}j_{1}}^{1}\ldots T_{j_{N-2}i_{N-1}}^{N-1}, \tag{2}\]

multiplying \(N\) elements of the tensors \(T^{i}\) to obtain the element of the tensor \(T\).

To exemplify the problem we want to solve, let's think about this. For a general case with bond dimension \(b\), the dimension of the index that is contracted between each pair of nodes, \(N\) nodes and \(a\) constant elements of the nodes we would have

\[T_{i_{0}i_{1}\ldots i_{N-1}}=a^{N}b^{N-1}. \tag{3}\]

We can see that with 20 nodes, an element value of 1.5 and a link dimension of 10, the final tensor elements would be 3.3 10\({}^{22}\). This is a very large element for a good initialization. However, if we were to divide the values of these tensors by that number, we could arrive at a case where \(a^{N}\) was a number too small for our computer to store and we would get a 0 in all the elements.

Footnote 1: We have to use the tensor network to compute the tensor network.

This problem is exacerbated by the number of nodes in the layer, since each one is a product. Moreover, we cannot simply calculate these tensor elements for cases with many physical indices, the output indices. This is because the number of values to be held in memory increases exponentially with the number of indices.

## 3 Tensor network initialization protocol

Our protocol is based on the use of partial Frobenius norms in order to normalize the total Frobenius norm of the resulting tensor.

The Frobenius norm of a matrix is given by the equation

\[||A||_{F}=\sqrt{\sum_{ij}|a_{ij}|^{2}}=\sqrt{\mathrm{Tr}(A^{\dagger}A)}. \tag{4}\]

In a tensor network, this would be to contract the layer with a copy of itself, so that each physical index is connected to the equivalent of its copy. We can see some examples in Fig. 3.

Figure 3: Square of the Frobenius norm calculated to a) Tensor Train layer. b) Tensor Train Matrix layer. c) PEPS layer.

Figure 2: a) Tensor Train layer with 5 indices. b) Tensor Train Matrix layer with 10 indices. c) PEPS layer with 9 indices.
The contraction of this tensor network is equivalent to the Frobenius norm of the matrix it represents. In addition, it can be computed without the need to calculate the elements of the represented matrix, using only the elements of the nodes.

The Frobenius norm is an indicator that serves to regularize layers of a model [5] and gives an estimate of the order of magnitude of the matrix elements. This can be seen if, with a more or less homogeneous distribution of elements, the norm in Eq. (4) will be of the order of \(\sqrt{nm}\ a_{00}\) for a \(n\times m\) matrix.

To avoid that the elements of the layer are too big or too small, and therefore we have too big or too small outputs in the initialization, we will normalize these elements so that the norm of the tensor is a number that we choose, for example 1. This prevents our highest element from being higher than this value, while taking advantage of the localized distribution of values to ensure that the smallest value is not too small.

Still, for an \(n\times m\) matrix we will be summing \(nm\) values, so we should adjust the norm to be proportional to \(nm\), the size of the problem, and thus not decrease the magnitude of the values with it.

For this purpose we define what we will call the partial square norm of the tensor network.

### Partial square norm of the tensor network

Throughout this section we will assume that we can consistently sort the nodes of a tensor network so that they form a single growing chain.

\(p||\mathcal{A}||_{n,N}\), the partial square norm at \(n\) nodes of a tensor network \(\mathcal{A}\) with N nodes, will be defined as the norm of the tensor network \(\mathcal{A}_{n}\) defined by the first \(n\) nodes of \(\mathcal{A}\).

To get an idea of what this partial square norm is, we will exemplify it with a simple case, a tensor train layer. We will consider the tensor network in Fig. 4, whose nodes are sorted.

As we can see, in this case we would only have to do the same process as when calculating the total norm of the total tensor network, but stopping at step \(n\) and contracting the bond index of the two final tensors of the chain.

We can see in the following Fig. 5 and 6 how the partial square norm would be for a TT-Matrix layer and for a PEPS layer.

This calculation can be extended to general tensor networks easily, as long as we have a consistent ordering of the nodes.

### Initialization protocol

If we have a tensor network \(\mathcal{A}\), representing a \(n_{A}\times m_{A}\) matrix whose Frobenius norm \(||\mathcal{A}||_{F}\) is infinite, zero or outside a certain range of values, we will want to normalize the elements of our tensor network so that the norm \(||\mathcal{B}||_{F}\) of the new tensor network \(\mathcal{B}\) is equal to a certain number. From here we will assume it is \(F=n_{A}m_{A}\), but we will see that another number can easily be chosen. To normalize the norm of the \(\mathcal{A}\) tensor with \(N\) nodes, we will only have to divide the elements of each of its nodes by \(||\mathcal{A}||_{F}^{1/N}\).

Since we cannot divide the elements by 0 or infinity, we will use the following logic. If the total norm is infinite (zero), there will exist a partial square norm of \(n\) nodes whose value is finite and non-zero such that the partial square norm of \(n+1\) nodes is infinite (zero). This is because each step we add a new node to the partial square norm, we multiply by a new value, so

Figure 4: a) Tensor Train layer with 5 nodes. b) Partial square norm at 1 node. c) Partial square norm at 2 nodes. d) Partial square norm at 3 nodes.

Figure 5: a) Tensor Train Matrix layer with 5 nodes. b) Partial square norm at 1 node. c) Partial square norm at 2 nodes. d) Partial square norm at 3 nodes.
infinity (zero) will appear after a certain number of nodes, being the partial square norm with one node less a valid number to divide by.

The idea is to iteratively normalize the norm little by little so that we eventually achieve full normalization.

We want a tensor network \(\mathcal{B}\) with Frobenius norm \(F\), with \(N\) nodes and we set as tolerance range \((a,b)\). The protocol to follow would be:

1. We initialize the node tensors with some initialization method. We recommend random initialization with a Gaussian distribution of a constant standard deviation (not greater than 0.5) and a constant mean neither too high nor too low and positive.
2. We calculate the norm \(||\mathcal{A}||_{F}\). If it is finite and non-zero, we divide each element of each node by \(\left(\frac{||\mathcal{A}||_{F}}{F}\right)^{1/N}\) and we have the \(\mathcal{B}\) we want. Otherwise, we continue.
3. We calculate \({}^{p}||\mathcal{A}||_{1,N}\), the partial square norm for 1 node of \(\mathcal{A}\). 1. If it is infinite, we divide each element of nodes of \(\mathcal{A}\) by \((10(1+\xi))^{1/2N}\), being \(\xi\) a random number between 0 and 1 and we return to step 2. 2. If it is zero, we divide each element of nodes of \(\mathcal{A}\) by \((0.1/(1+\xi))^{1/2N}\) and we return to step 2. 3. Otherwise, we save this value as \({}^{p}||\mathcal{A}||_{1,N}\) and continue.
4. For \(n\in[2,N-1]\) we calculate \({}^{p}||\mathcal{A}||_{n,N}\), the partial square norm for \(n\) nodes of \(\mathcal{A}\). 1. If it is infinite or zero, we divide each element of nodes of \(\mathcal{A}\) by \((^{p}||\mathcal{A}||_{n-1,N})^{\frac{1}{2N}}\) and we return to step 2. 2. If it is finite, but bigger than \(b\) or smaller than \(a\), we divide each element of nodes of \(\mathcal{A}\) by \((^{p}||\mathcal{A}||_{n,N})^{\frac{1}{2N}}\) and we return to step 2. 3. Otherwise, we continue.
5. If no partial square norm is out of range, infinite or zero, we divide each element of nodes of \(\mathcal{A}\) by \((^{p}||\mathcal{A}||_{N-1,N})^{\frac{1}{2N}}\) and we return to step 2.

We repeat the cycle until we reach a stop condition, which will be to have repeated a certain maximum number of iterations. If we reach that point, the protocol will have failed and we will have two options. The first is to change the order of the nodes, so that other structures are checked. The second is to reinitialize with other hyperparameters in the nodes.

The purpose of using a random factor in case of divergence in the partial norm with 1 node is that, not knowing the real value by which we should divide, we rescale by an order of magnitude. However, to avoid possible infinite rescaling loops, we add a variability factor so that we cannot get stuck.

## 4 Results

We ran the initialization with TT and TT-Matrix layers of different sizes and physical dimensions \(p\), and checked how many steps were needed for normalization. We use a value of 1 for the mean and a value of 0.5 for the standard deviation. For the TT layer we choose \(F=p^{2N}\) and for the TT-Matrix layer we choose \(F=p^{N}\). For the TT layer this is the number of elements we have and for the TT-Matrix layer it is its root, a number we take for convergence purposes. Our tolerance range is \((F\ 10^{-3},F\ 10^{3})\). We can see in Figs. 7, 8 and 9 the result for the TT layer and for the TT-Matrix layer.

We can see in Fig. 7 that the scaling with \(N\) is linear for different \(p\). Fig. 8 shows that the scaling is logarithmic with \(p\), similar to the scaling with \(b\) in Fig. 9. In all cases, the TT-Matrix layer normalization requires more steps than the TT layer normalization.

Figure 6: a) PEPS layer with 9 nodes. b) Partial square norm at 1 node. c) Partial square norm at 2 nodes. d) Partial square norm at 5 nodes.
## 5 Other applications

So far we have seen the application for tensorized neural networks, but this method can be useful for more methods.

Whenever we have a method where we have to contract a tensor network and the non-zero elements of the tensors that form it are of the same order of magnitude we can perform this method. This can be helpful in cases where we do not want the absolute scale of the final tensor elements, but we want to observe a relative scale between them.

An example would be the simulation of imaginary time evolution processes where we want to see which is the state with minimum energy and not the energy it has. However, this energy could be recovered if we perform the method, we save the scale factor by which we are multiplying the elements of the tensor network, and we multiply the values of the resulting tensor network by this factor. This can be interesting because the different factors can be multiplied keeping their order of magnitude apart, so that we do not have overflows.

## 6 Conclusions

We have developed a method to successfully initialize a layer of tensorized neural networks by using the partial computations of their Frobenius norms. We have also applied it to different layers and seen its scaling.

A possible future line of research could be to investigate how to reduce the number of steps to be performed. Another could be to study the scaling of complexity with increasing size of each of the different types of existing layers. We could also apply it to the methods mentioned in Sec. 5 for example in combinatorial optimization[6] to determine the appropriate decay factor.

## Acknowledgement

The research leading to this paper has received funding from the Q4Real project (Quantum Computing for Real Industries), HAZITEK 2022, no. ZE-2022/00033.
