Compressing neural network by tensor network with exponentially fewer
  variational parameters
Neural network (NN) designed for challenging machine learning tasks is in
general a highly nonlinear mapping that contains massive variational
parameters. High complexity of NN, if unbounded or unconstrained, might
unpredictably cause severe issues including over-fitting, loss of
generalization power, and unbearable cost of hardware. In this work, we propose
a general compression scheme that significantly reduces the variational
parameters of NN by encoding them to deep automatically-differentiable tensor
network (ADTN) that contains exponentially-fewer free parameters. Superior
compression performance of our scheme is demonstrated on several
widely-recognized NN's (FC-2, LeNet-5, AlextNet, ZFNet and VGG-16) and datasets
(MNIST, CIFAR-10 and CIFAR-100). For instance, we compress two linear layers in
VGG-16 with approximately $10^{7}$ parameters to two ADTN's with just 424
parameters, where the testing accuracy on CIFAR-10 is improved from $90.17 \%$
to $91.74\%$. Our work suggests TN as an exceptionally efficient mathematical
structure for representing the variational parameters of NN's, which exhibits
superior compressibility over the commonly-used matrices and multi-way arrays.
Yong Qing, Ke Li, Peng-Fei Zhou, Shi-Ju Ran
# Compressing neural network by tensor network with exponentially fewer variational parameters

###### Abstract

Neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains massive variational parameters. High complexity of NN, if unbounded or unconstrained, might unpredictably cause severe issues including over-fitting, loss of generalization power, and unbearable cost of hardware. In this work, we propose a general compression scheme that significantly reduces the variational parameters of NN by encoding them to multi-layer tensor networks (TN's) that contain exponentially-fewer free parameters. Superior compression performance of our scheme is demonstrated on several widely-recognized NN's (FC-2, LeNet-5, and VGG-16) and datasets (MNIST and CIFAR-10), surpassing the state-of-the-art method based on shallow tensor networks. For instance, about 10 million parameters in the three convolutional layers of VGG-16 are compressed in TN's with just \(632\) parameters, while the testing accuracy on CIFAR-10 is surprisingly improved from \(81.14\%\) by the original NN to \(84.36\%\) after compression. Our work suggests TN as an exceptionally efficient mathematical structure for representing the variational parameters of NN's, which superiorly exploits the compressibility than the simple multi-way arrays.

 neural network, tensor network, machine learning, model compression, tensor decomposition.

## I Introduction

Neural network (NN) [1] has gained astounding success in a wide range of fields including computer vision, natural language processing, and most recently scientific investigations in, e.g., applied mathematics(e.g., [2, 3]) and physics(e.g., [4, 5, 6, 7, 8, 9, 10]). To improve the performance on handling complicated realistic tasks, the amount of variational parameters in NN rapidly increases from millions to trillions (e.g., Chat-GPT with 175 billion parameters [11]). Such a paradigm with the utilization of super large-scale NN's brought us several severe challenges. Though the representation ability of NN should be enhanced by increasing the complexity, over-fitting might occur and the generalization ability of NN might unpredictably be harmed. The increasing complexity also brings unbearable cost of hardware in academic investigations and practical applications.

The variational parameters in NN are usually stored as high-order tensors (or multi-linear arrays). Previous results show that generalization ability can be improved by suppressing or "refining" the degrees of freedom in such tensors by, e.g., network pruning [12, 13, 14], knowledge distillation [15, 16], weight sharing [17, 18], tensor decompositions [19, 20], etc. In recent years, shallow tensor networks (TN's) including matrix product state (MPS [21, 22], also known as tensor-train form [23]) and matrix product operator (MPO [24]) were applied to represent the tensors in neural networks, and achieved remarkable compression rates [25, 26, 27, 28, 29]. These inspiring successes strongly imply the existence of efficient mathematical structures over the simple multi-linear arrays for representing the variational parameters of NN's, which yet remains elusive.

In this work, we propose to adopt multi-layer TN [30] as an exceptionally efficient representation of variational parameters to fully exploit the compressibility of NN's. Our idea (illustrated in Fig. 1) is to encode the variational parameters of a considered NN layer into the contraction of a TN that contains exponentially fewer parameters. For instance, the number of parameters of the TN encoding \(2^{Q}\) parameters of NN just linearly as \(O(MQ)\), with \(M\sim O(1)\) the number of TN layers. Since the contraction process is differentiable, automatic differentiation technique [31, 32] is utilized to obtain the tensors of the TN to reach the optimal accuracy after compression. Thus we dub our scheme as automatically differentiable tensor network (ADTN).

The compression performance and generality of ADTN are demonstrated on various well-known NN's (FC-2, LeNet

Fig. 1: (Color online) The workflow of ADTN for compressing NN. (a) The illustration of a convolutional NN as an example, whose variational parameters (\(\mathbf{T}\)) are encoded in a ADTN shown in (b). The contraction of the ADTN results in \(\mathbf{T}\), in other words, where the ADTN contains much less parameters than \(\mathbf{T}\). In (c), we show the diagrammatic representation of the unitary conditions for the tensors in ADQC.
5 [33], and VGG-16 [34]) and datasets (MNIST [35] and CIFAR-10 [36]). For example, about \(10^{7}\) parameters in three convolutional layers of VGG-16 are compressed in three ADTNs with just \(632\) parameters, while the testing accuracy on CIFAR-10 is surprisingly improved from \(81.14\%\) of the original NN to \(84.36\%\) after compression. Such compression efficiency surpasses the state-of-the-art MPO method. Finally, we impose unitary conditions on the tensors of ADTN and propose automatically differentiable quantum circuit (ADQC [32]) to further reduce the parameter complexity. Our work implies a general way to reduce the excessive complexity of large-scale NN's by encoding the variational parameter into more compact and efficient mathematical form.

## II Method

A NN can be formally written as a mapping

\[\mathbf{y}=f(\mathbf{x};\mathbf{T},\mathbf{T}^{\prime},\ldots), \tag{1}\]

which maps the input features \(\mathbf{x}\) to the output vector \(\mathbf{y}\). We aim to compress the variational parameters \((\mathbf{T},\mathbf{T}^{\prime},\ldots)\) or a major part of them by ADTN. Note we do not assume the types of layers (linear, convolutional, etc.) in the NN as it makes no difference to our ADTN method.

Denoting the parameters to be compressed as tensor \(\mathbf{T}\), our idea is to represent \(\mathbf{T}\) as the contraction of ADTN with multiple layers. The structure of ADTN is flexible. Here, we choose the commonly used "brick-wall" structure; see an example of its diagrammatic representation in Fig. 1 (b). In simple words about the diagrammatic representation of TN, each block represents a tensor, and the bonds connected to a block represent the indexes of the corresponding tensor. Shared bonds represent the dumb indexes that should be summed up (contracted). The vertical black lines represent the activation function (e.g., ReLU). More details about TN contractions and the diagrammatic representation can be found in the Appendix B or Ref. [30].

For simplicity, we assume to compress \(2^{Q}\) variational parameters, i.e., \(\#(\mathbf{T})=2^{Q}\). Obviously, the contraction of ADTN should also give us a tensor (denoted as \(\mathbf{\mathcal{T}}\)) containing \(2^{Q}\) parameters. In our example, the ADTN results in a \(Q\)-th order tensor, where we take the dimension of each index as 2. These indexes are represented by the unshared bonds in the ADTN. See the blue bonds on the right boundary of the diagrammatic representation in Fig. 1 (b).

The main part of this exemplified ADTN is formed by the \((d\times d\times d\times d)\) tensors \(\{\mathbf{A}^{[k]}\}\) (\(k=0,\ldots,K-1\) with \(K\) their total number), which form the variational parameters of ADTN. We here take \(d=2\). On the left boundary of ADTN, we put several two-dimensional constant vectors \(\mathbf{v}\) for the convenience in formulating and analyzing. We may take \(\mathbf{v}=[1,0]\) for instance.

For the brick-wall structure, we define each \((Q-1)\) tensors in two neighboring columns [depicted in Fig. 1 (b)] as a TN layer. Contracting the tensors in one tensor layer essentially gives us a (\(2^{Q}\times 2^{Q}\)) matrix denoted as \(\mathbf{\mathcal{L}}\). The mapping represented by the whole ADTN can be written as

\[\mathbf{\mathcal{T}}=\mathbf{\mathcal{L}}\sigma(\ldots\mathbf{\mathcal{L}}\sigma(\mathbf{ \mathcal{L}}\sigma(\mathbf{\mathcal{L}}\prod_{\otimes q=1}^{Q}\mathbf{v}))), \tag{2}\]

with \(\otimes\) the tensor product and \(\sigma\) the activation function. In other words, \(\mathbf{\mathcal{T}}\) is obtained by implementing the mapping \(\sigma(\mathbf{\mathcal{L}}(*))\) for \(M\) times with \(M\) the number of TN layers. Be aware that one does not have to simulate the matrix \(\mathbf{\mathcal{L}}\), which is introduced here only for the convenience of understanding and expressing. The total number of variational parameters in such a brick-wall ADTN scales as

\[\#(\text{ADTN})\equiv\sum_{k}\#(\mathbf{A}^{[k]})\sim O(MQ), \tag{3}\]

with \(M\sim O(1)\) according to our results provided below. In plain words, we encode \(2^{Q}\) parameters in an ADTN that contains exponentially-fewer parameters.

There are two main stages to obtain \(\{\mathbf{A}^{[k]}\}\). The first is pretraining by minimizing the distance between \(\mathbf{T}\) and \(\mathbf{\mathcal{T}}\). We choose the loss function as Euclidean distance \(L^{\text{E}}=|\mathbf{\mathcal{T}}-\mathbf{\mathcal{T}}|\). After \(L^{\text{E}}\) converges, we start the second stage to minimize the loss function for implementing the machine learning task. One may choose the same loss function as the one used for training the NN, such as the cross entropy from the outputs of NN and the labels of the images' classifications. The feed-forward process is also the same as that for training the NN, except that the parameters (\(\mathbf{T}\)) of the target NN layer are replaced by the tensor (\(\mathbf{\mathcal{T}}\)) obtained by contracting the ADTN. Generally speaking, the first stage is to enhance the stability by finding a proper initialization of ADTN, and the second stage further improves the performance with an end-to-end optimization. The mapping given by NN in the second stage can be written as

\[\mathbf{y}=f(\mathbf{x};\{\mathbf{A}^{[k]}\},\mathbf{T}^{\text{res}}), \tag{4}\]

with \(\mathbf{T}^{\text{res}}\) denoting the uncompressed parameters.

The parameter complexity of ADTN can be further reduced by imposing unitary conditions on \(\{\mathbf{A}^{[k]}\}\). See the diagrammatic representation of the unitary condition in Fig. 1 (c). The ADTN formed by unitary tensors without activation functions is called ADQC [32], which was originally proposed for quantum state preparation [37]. The contraction of ADQC results in a unitary that "evolves the initial product state" \(\prod_{\otimes}\mathbf{v}\) to the final entangled state \(\mathbf{\mathcal{T}}\), in the terminology of quantum physics.

ADQC is expected to also possess generality for compressing NN thanks to its remarkable power on preparing a wide range of quantum states. One just needs to take care of the norm since unitary transformation preserves the norm, i.e., \(|\prod_{\otimes}\mathbf{v}|=|\mathbf{\mathcal{T}}|\). In our example where the dimension of tensors in ADTN is taken as \((2\times 2\times 2\times 2)\), the unitary conditions introduce additionally ten constraints for each tensor and reduce its number of free parameters from \(2^{4}\) to \(6=(16-10)\).

## III Results

Table I shows the performance of ADTN with \(M=1\) to compress various NN's. Our first example is FC-2 formed
by two linear layers, whose sizes are taken as \((784\times 256)\) and \((256\times 10)\), respectively (see Table III). We slice out \(2^{17}\) parameters from the first linear layer and compress them in an ADTN with 160 parameters. The testing accuracy on MNIST remains almost unchanged after compression.

Our second example is LeNet-5, which is a convolutional NN that contains three convolutional and two linear layers (see Table IV). Though LeNet-5 has much fewer parameters than FC-2, the more advanced and compact structure of the convolutional layers makes LeNet-5 to outperform FC-2 on processing images. We take \(2^{13}\) weights in the third convolutional layer and \(2^{15}\) weights in the first linear layer, which are respectively compressed to two ADTN's with in total 260 parameters. Still, the compression by ADTN does not cause any harm to the testing accuracy of LeNet-5, though LeNet-5 is already much more compact than FC-2.

Then we try to compress VGG-16, which is a much more complicated convolutional NN with thirteen convolutional and three linear layers (see Table V). We use three ADTNs with 632 parameters in total to compress \(2^{21}+2^{22}+2^{22}\simeq 10^{7}\) parameters in the last three convolutional layers. Amazingly, the testing accuracy increases for about \(2.2\%\) after such a significant compression.

By imposing unitary conditions, ADQC further reduces the number of parameters compared with ADTN, and manages to keep the testing accuracy unharmed compared with the original NN's. This demonstrates the astounding power of ADQC for squeezing out the parameter redundancy of NN.

We compare ADTN with the state-of-the-art compression scheme based on MPO [28, 29]. It is a mathematical structure generalized from the famous MPS [21, 22], and thus belongs to the shallow TN's. In contract, ADTN contains multiple layers of tensors, and belongs to the deep TN's. The testing accuracy \(\eta\) on CIFAR-10 after compressing the last three convolutional layers in VGG-16 versus the inverse of compression ratio \(1/\rho\) is shown in Fig. 2, where the compression ratio is defined as

\[\rho=\frac{\#(\text{ADTN})}{\#(\mathbf{T})}. \tag{5}\]

We vary the number of TN layers in ADTN from \(M=1\) to \(5\) to obtain different values of \(\rho\). Both ADTN and MPO improve the testing accuracy of the uncompressed NN (with \(\eta_{\text{NN}}=0.8114\)). ADTN demonstrates higher compression efficiency than MPO. For instance, to reach the test accuracy \(\eta\simeq 84\%\), MPO can reduce the number of parameters by about \(1/\rho\simeq 2000\) times, while ADTN reduces by more than \(1/\rho\simeq 17000\) times. For a similar compression ratio, say \(1/\rho\simeq 2000\) (see the vertical dash line), ADTN achieve higher testing accuracy (\(\simeq 0.857\)) than MPO (\(\simeq 0.833\)).

The superior performance of ADTN over MPO is consistent with the results for implementing quantum state preparation. In Ref. [32], the authors concluded that by writing an MPS to an ADQC acted on a trivial product state, the parameter complexity of the MPS can be significantly reduced, where the compression ratio approximately scales \(\rho\sim O(M\chi^{-2})\) with \(\chi\) a hyper-parameter of MPS to control its complexity. Thus, ADQC can compress the parameter complexity of MPS in most cases. The structure of MPO is essentially the same as MPS. It is reasonable that ADTN can compress MPO itself and thus surpass MPO for compressing NN's.

We further test ADTN by compressing different layers in VGG-16. The first column in Table II illustrates the amount of parameters in each layers, and the compressed ones are marked in red. In the first two cases where we compress the two or three layers with the highest parameter complexity, the training accuracy is almost \(100\%\) and the testing accuracy surpasses

\begin{table}
\begin{tabular}{|c|c c c c|c c c c c|} \hline  & & \multicolumn{3}{c|}{**NN**} & \multicolumn{3}{c|}{**ADTN \& ADQC**} \\ \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{Model} & Testing & Total & Parameters & \multirow{2}{*}{\begin{tabular}{c} \# Compressed \\ layers (\(N\)) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} \# \\ unitary \\ (\(N\)) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Total \\ parameters \\ (\(N\)) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Testing \\ parameters \\ (\(N\)) \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} Compression \\ accuracy \\ (\(N\)) \\ \end{tabular} } \\ \hline \multirow{4}{*}{**MNIST**} & FC-2 & 97.94\% & 203530 & 131072 & No & 160 & 97.81\% & 99.69\% & \(1.2\times 10^{-3}\) \\  & & & & Yes & 80 & 97.79\% & 99.75\% & \(6.1\times 10^{-4}\) \\ \cline{2-10}  & \multirow{2}{*}{LeNet-5} & \multirow{2}{*}{99.15\%} & \multirow{2}{*}{61706} & \multirow{2}{*}{40960} & No & 260 & 99.43\% & 100.25\% & \(6.4\times 10^{-3}\) \\  & & & & Yes & 130 & 99.40\% & 100.28\% & \(3.2\times 10^{-3}\) \\ \hline \multirow{2}{*}{**CIFAR-10**} & \multirow{2}{*}{VGG-16} & \multirow{2}{*}{81.14\%} & \multirow{2}{*}{14542474} & No & 632 & 84.36\% & 103.97\% & \(6.0\times 10^{-5}\) \\  & & & & Yes & 312 & 81.47\% & 100.40\% & \(3.0\times 10^{-5}\) \\ \hline \end{tabular}
\end{table} TABLE I: The performance (testing accuracy, compression radio, etc.) of ADTN and ADQC on compressing FC-2, LeNet-5 [33], and VGG-16 [34] trained for classifying the images in MNIST [35] and CIFAR-10 [36].

Fig. 2: (Color online) The testing accuracy \(\eta\) on CIFAR-10 of VGG-16 after compression versus the inverse of compression ratio (\(1/\rho\)). The last three convolutional layers are compressed by ADTN or MPO [28], respectively. The testing accuracy of VGG-16 before compression is \(\eta_{\text{NN}}=0.8114\), indicated by a horizontal dash line.
that of the original NN (\(\eta_{\text{NN}}=81.14\%\)). The ADTN with one TN layer (\(M=1\)) is sufficient to compress a minority of NN layers without harming the accuracies.

In the last column, we show the total compression ratio

\[\rho_{\text{tot}}=\frac{\#(\text{ADTN})+\#(\mathbf{T^{\text{res}}})}{\#(\text{NN})} \simeq\frac{\#(\mathbf{T^{\text{res}}})}{\#(\text{NN})}. \tag{6}\]

The approximation holds since in general we have \(\#(\text{ADTN})\ll\#(\mathbf{T^{\text{res}}})\ll\#(\text{NN})\). To further reduce \(\rho_{\text{tot}}\), one may consider to compress more layers by ADTN. In such cases, multiple TN layers are required to improve training and testing accuracies. For the example, to compress 12 layers in VGG-16, the testing accuracy is improved from \(72.13\%\) to \(78.95\%\) by increasing \(M\) from \(1\) to \(9\). The total compression ratio reaches \(\rho_{\text{tot}}\simeq 0.008\) (reducing for about \(1/\rho_{\text{tot}}\simeq 125\) times).

We finally compare the testing accuracies and (inverse of) compression ratios of ADTN and ADQC (Fig. 3). While all accuracies surpass that of the original VGG-16, ADTN achieves better accuracy than ADQC. This implies the additional compression by introducing unitary conditions will generally lower the accuracy (to a tolerable extent). Despite this, the validity of ADQC for compressing sheds light on applying quantum computing for compression. One just needs to restore all the parameters of the unitary tensors (quantum gates), and implement them on the initial product state \(\prod_{\otimes=1}^{Q}\mathbf{v}\). The tensor \(\mathbf{\mathcal{T}}\) can be reached by the quantum state tomography on the output state.

## IV Summary

In summary, we propose a general compression scheme to significantly reduces the variational parameters of neural networks (NN's). The key idea is to restore the high-order tensors that stores the variational parameters of NN by the contractions of multi-layer tensor networks (TN's), which contain exponentially less parameters than the high-order tensors. The performance of our scheme has been demonstrated on several well-known NN's including simple fully-connected NN and large-scale convolutional NN's. Our work suggest TN as a more efficient and compact mathematical form than high-order tensors to represent the variational parameters of NN's.

## Acknowledgments

We are indebted to Ze-Feng Gao for stimulative discussions. This work was supported by NSFC (Grant No. 12004266 and No. 11834014), Beijing Natural Science Foundation (Grant No. 1232025), Foundation of Beijing Education Committees (No. KM202010028013), and by the Academy for Multidisciplinary Studies, Capital Normal University.

## Appendix A Details of FC-2, LeNet-5, and VGG-16

Below, we provide the structure and hyper-parameters of the neural networks (FC2 in Table III, LeNet-5 in Table IV, and VGG-16 in Table V), for the convenience of reproducing our results.

\begin{table}
\begin{tabular}{c|c c c c c} \hline \begin{tabular}{c} Compressed \\ layers \\ _(and)_ \\ \end{tabular} & \begin{tabular}{c} Training \\ accuracy \\ (\(\bar{\eta}\)) \\ \end{tabular} & \begin{tabular}{c} Testing \\ accuracy \\ \end{tabular} & \begin{tabular}{c} Compression \\ \(\eta/\eta_{\text{NN}}\) \\ (\(\rho\)) \\ \end{tabular} & \begin{tabular}{c} Total \\ compression \\ ratio (\(\rho_{\text{tot}}\)) \\ \end{tabular} \\ \hline \multirow{4}{*}{\begin{tabular}{c} 1 \\ 3 \\ 5 \\ 5 \\ 100\% \\ \end{tabular} } & 19 & 98.9\% & 84.2\% & 103.86\% & \(5.0\times 10^{-5}\) \\  & 3 & 100\% & 85.5\% & 105.63\% & \(3.7\times 10^{-4}\) \\  & 5 & 100\% & 85.1\% & 105.63\% & \(3.7\times 10^{-4}\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} 1 \\ 3 \\ 5 \\ 5 \\ 5 \\ 5 \\ 5 \\ 5 \\ 5 \\ 5 \\ 5 \\ 7 \\ \end{tabular} } & 19 & 99.9\% & 84.8\% & 103.9\% & 6.0 \(\times 10^{-5}\) \\  & 3 & 100\% & 85.1\% & 105.00\% & \(2.5\times 10^{-4}\) \\  & 5 & 100\% & 85.6\% & 105.66\% & \(4.0\times 10^{-4}\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} 1 \\ 3 \\ 5 \\ 5 \\ 5 \\ 7 \\ 81.55\% \\ 9 \\ 81.95\% \\ \end{tabular} } & 19 & 98.8\% & 78.5\% & 96.7\% & 8.0 \(\times 10^{-5}\) \\  & 3 & 99.9\% & 83.2\% & 101.60\% & \(3.3\times 10^{-4}\) \\  & 5 & 100\% & 82.7\% & 102.02\% & \(5.9\times 10^{-4}\) \\ \hline \multirow{4}{*}{\begin{tabular}{c} 1 \\ 5 \\ 7 \\ 81.95\% \\ 9 \\ \end{tabular} } & 19 & 96.3\% & 65.93\% & 81.25\% & 1.0 \(\times 10^{-4}\) \\  & 3 & 99.94\% & 77.73\% & 95.78\% & \(4.2\times 10^{-4}\) \\  & 5 & 100\% & 79.15\% & 95.57\% & \(7.4\times 10^{-4}\) \\  & 7 & 100\% & 80.61\% & 99.35\% & \(1.1\times 10^{-3}\) \\ \hline \multirow{4}{*}{
\begin{tabular}{c} 3 \\ 5 \\ 7 \\ 9 \\ 81.95\% \\ \end{tabular} } & 37 & 75.78\% & 72.13\% & 88.90\% & \(1.0\times 10^{-4}\) \\  & 5 & 78.13\% & 74.24\% & 91.50\% & \(4.2\times 10^{-4}\) \\  & 7 & 81.54\% & 77.61\% & 95.65\% & \(7.4\times 10^{-4}\) \\  & 9 & 81.95\% & 78.59\% & 96.86\% & \(1.1\times 10^{-3}\) \\ \hline \end{tabular}
\end{table} TABLE II: The performance of ADTN on VGG-16 by compressing different NN layers. The parameter complexities of different layers of VGG-16 are illustrated by the lengths of bars in the first column, where the compressed ones are marked in red. The third and forth columns show the training and testing accuracies after compressing by ADTN with \(M\) TN layers. The fifth column show the testing accuracy after compression divided by that of the original VGG-16. The sixth and seventh columns show compression ratios for different layers and the total compression ratio [Eq. (6)].

Fig. 3: (Color online) The inverse of compression ratio \(1/\rho\) (left y-axis) and testing accuracy \(\eta\) (right y-axis) of compressing VGG-16 with ADTN and ADQC.
To explain how the high-order tensor \(\mathbf{\mathcal{T}}\) by contracting the ADTN, we show in Fig. 4 (a) and (b) the diagrammatic representations of two local contractions as an example. As shown in (a), the contraction of one tensor \(\mathbf{A}^{[0]}\) in ADTN and two vectors \(\mathbf{v}\) reads

\[u_{cd}^{[0]}=\sum_{ab}v_{a}v_{b}A_{abcd}^{[0]}. \tag{7}\]

Another example is the contraction shown in (b), which reads

\[w_{aefd}=\sum_{bc}u_{ab}^{[0]}u_{cd}^{[1]}A_{bcef}^{[3]}, \tag{8}\]

where \(\mathbf{u}^{[1]}\) denotes the contraction of \(\mathbf{A}^{[1]}\) and two \(\mathbf{v}\)'s. Generally speaking, the contraction formulas and their diagrammatic representations are in one-to-one correspondence. After contracting all shared bonds (indexes) by following the same rules as Eqs. (7) and (8), the ADTN results in a high-order tensor \(\mathbf{\mathcal{T}}\) that represents the variational parameters of NN, whose indexes are denoted by the blue bonds in (c).

With the presence of activations, the contractions should be done layer by layer, from left to right. After contracting one TN layer, each element of the obtained tensor will be put into the activation function. In the simulations, we take the ReLU activation, where an element \(x\) is mapped as

\[\sigma(x)=\max(0,x). \tag{9}\]
