Tensor Networks Meet Neural Networks: A Survey and Future Perspectives
Tensor networks (TNs) and neural networks (NNs) are two fundamental data
modeling approaches. TNs were introduced to solve the curse of dimensionality
in large-scale tensors by converting an exponential number of dimensions to
polynomial complexity. As a result, they have attracted significant attention
in the fields of quantum physics and machine learning. Meanwhile, NNs have
displayed exceptional performance in various applications, e.g., computer
vision, natural language processing, and robotics research. Interestingly,
although these two types of networks originate from different observations,
they are inherently linked through the common multilinearity structure
underlying both TNs and NNs, thereby motivating a significant number of
intellectual developments regarding combinations of TNs and NNs. In this paper,
we refer to these combinations as tensorial neural networks (TNNs), and present
an introduction to TNNs in three primary aspects: network compression,
information fusion, and quantum circuit simulation. Furthermore, this survey
also explores methods for improving TNNs, examines flexible toolboxes for
implementing TNNs, and documents TNN development while highlighting potential
future directions. To the best of our knowledge, this is the first
comprehensive survey that bridges the connections among NNs, TNs, and quantum
circuits. We provide a curated list of TNNs at
\url{https://github.com/tnbar/awesome-tensorial-neural-networks}.
Maolin Wang, Yu Pan, Zenglin Xu, Xiangli Yang, Guangxi Li, Andrzej Cichocki
# Tensor Networks Meet Neural Networks:

###### Abstract

Tensor networks (TNs) and neural networks (NNs) are two fundamental types of data modeling approaches. TNs have been proposed as a solution to the curse of dimensionality faced by large-scale tensors by converting an exponential number of dimensions to polynomial complexity. Thus, they have attracted many studies in the fields of quantum physics and machine learning. On the other hand, NNs are computing systems inspired by the biological NNs that constitute human brains. Recently, NNs and their variants have achieved outstanding performance in various applications, e.g., computer vision, natural language processing, and robotics research. Interestingly, although these two types of networks come from different observations, they are inextricably linked via the common intrinsic multilinearity structure underlying both TNs and NNs. Consequently, a significant number of intellectual sparks regarding combinations of TNs and NNs have burst out. The combinations described as "tensor networks meet neural networks" are termed tensorial neural networks (TNNs) in this paper. This survey introduces TNNs based on three aspects. 1) Network Compression. TNs can greatly reduce parameters in NNs and satisfy the idea of constructing effective NNs. 2) Information Fusion. This can naturally and effectively enhance NNs with their ability to model the interactions among multiple modalities, views, or sources of various data. 3) Quantum Circuit Simulation. TNs can assist in designing and simulating quantum neural networks (ONNs). This survey also investigates methods for improving TNNs, examines useful toolboxes for implementing TNNs, and attempts to document TNN development and highlight its potential future directions. To the best of our knowledge, this is the first comprehensive survey to bridge the connections among NNs, TNs, and quantum circuits. We provide a curated list of TNNs at [https://github.com/tnbar/awesome-tensorial-neural-networks](https://github.com/tnbar/awesome-tensorial-neural-networks).

 Tensor Networks, Neural Networks, Network Compression, Information Fusion, Quantum Circuit Simulation

## 1 Introduction

Tensors are high-order arrays that represent the multiway interactions among multiple modal sources. In contrast, vectors (i.e., first-order tensors) and matrices (i.e., second-order tensors) are accessed in only one or two modes, respectively. As a common data type, tensors have been widely observed in several scenarios [1, 2, 3, 4]. For instance, functional magnetic resonance imaging (fMRI) samples are inherently fourth-order tensors that are composed of three-dimensional voxels that change over time [5, 6, 7]. In quantum physics, variational wave functions used to study many-body quantum systems are also high-order tensors [8, 9]. For spatiotemporal traffic analysis, road flow/speed information, which is collected from multiple roads over several weeks, can also be structured as a third-order tensor (road segment\(\times\)day\(\times\)time of day) [10]. However, for higher-order tensors, when the number of modes increases, the total number of elements in the tensors increases exponentially, resulting in a catastrophe when storing and processing tensors. Such a phenomenon is also recognized as the "curse of dimensionality" [11].

**Tensor Networks (TNs).** TNs [8, 11, 12] are generally countable collections of small-scale tensors that are interconnected by tensor contractions. These small-scale tensors are referred to as "components", "blocks", "factors", or "cores". Very large-scale tensors can be approximately represented in extremely compressed and distributed formats through TNs. Thus, it is feasible to implement distributed storage and efficient processing for high-order tensors that could not be dealt with before. By using TN methods, the curse of dimensionality can be alleviated or completely overcome [11]. Commonly used TN formats include CANDECOMP/PARAFAC (CP) [13, 14, 15], Tucker decomposition [16, 17], Blockerm Tucker (BTT) decomposition [18, 19, 20], Matrix Product State (MPS)/Tensor Train (TT) decomposition [21, 22, 23, 24], Matrix Product Operators (MPO)/matrix Tensor Train (mTT) decomposition [21, 22, 23, 24], Tensor Ring (TR) decomposition [25], Tree TN/Hierarchical Tucker (HT) decomposition [26], Projected Entangled Pair State (PEPS)/Tensor Grid decomposition [8, 27, 28], Multiscale Entanglement Renormalization [29], etc. For the purpose of understanding the interconnected structures of TNs, a TN diagram was developed as a straightforward graphical diagram (which is discussed in Section 2.2). A TN can provide a theoretical and computational framework for the analysis of some computationally unacceptable tasks. For example, based on the low-rank structures of TNs, Pan et al. [30] were able to solve the quantum random circuit sampling problem in 15 hours using 512 graphics processing units (GPUs); this problem was previously believed to require over 10,000 years on the most powerful classic electronic supercomputer and effectively challenge the quantum supremacy of Google's quantum computer called "Sycamore". Other applications include brain analysis [31], quantum chemistry calculation [32], human face clustering [33], dimensionality reduction [34], missing value estimation [35], latent factor analysis [36], subspace learning [37], etc.
**Neural Networks (NNs).** NNs are biologically inspired learning paradigms that enable a machine to learn knowledge from observational data through backpropagation [38, 39]. NNs that are stacked with multiple layers, i.e., deep NNs (DNNs) [40, 41], are widely used in the field of artificial intelligence due to their powerful ability to capture abundant information from deep structures. Typical types of DNNs include restricted Boltzmann machines (RBMs) [42], convolutional NNs (CNNs) [41, 43], recurrent NNs (RNNs) [44, 45], and Transformers [46, 47]. DNNs currently reach state-of-the-art performance in a wide range of applications in computer vision [48] and natural language processing [49]. For example, a number of CNN architectures such as AlexNet [50], VGGNet [51], GoogLeNet [52] and ResNet [53] won championships on the ImageNet dataset [54], demonstrating good potential for solving image classification tasks. Particularly, Alphafold [55, 56], which is a kind of Transformer architecture, can identify the structure of a protein in days, which previously took researchers years. Recently, Alphafold2 [55, 56] predicted the structures of nearly all known proteins with average atomic precision. Deep learning techniques are still pushing forward advances in a number of disciplines, including speech recognition [57], DNA mutations detection [58], structural biology [55, 56], drug discovery [59], food security [60], etc.

**Tensor Networks Meet Neural Networks.** TNs and NNs are two types of networks that come from different origins and have achieved success from different aspects, as mentioned above. Interestingly, they are closely bonded through their multilinear mathematical property rather than being orthogonal to each other [11]. Therefore, a promising approach is to integrate them via multilinearity to attain the objective that "the whole is greater than the sum of the parts." The main advantages of TNs are their compact structures, multiple entries, and close relationships with quantum mechanics, while NNs are well known for their wide applications [8, 12]. Based on these observations, it is feasible to combine TNs and NNs in three ways.

(1) Network Compression. NNs have achieved many successes in various tasks [40, 41, 41]. However, NNs still suffer from excess linear product calculations with massive numbers of dimensions and the curse of dimensionality [78]. A promising solution for addressing this issue is to utilize the lightweight and multilinear characteristics of TNs [78, 68, 79]. In detail, TNs can decompose any tensor of NNs into smaller blocks, thereby reducing the dimensionality to linear complexity [61, 62]. For example, in comparison to utilizing naive long short-term memory (LSTM) for action recognition tasks, TR-LSTM [79] models, which leverage TN technology to decompose weight tensors, can

\begin{table}
\begin{tabular}{c c l l c} \hline \hline \multicolumn{1}{c}{**Category**} & \multicolumn{1}{c}{**Subcategory**} & \multicolumn{1}{c}{**Detailed Models/Techniques**} & \multicolumn{1}{c}{**Section**} \\ \hline \hline \multirow{8}{*}{**TNNs**} & \multirow{3}{*}{Convolutional Neural Networks} & CP-CNN [61, 62, 63], Tucker-CNN [64, 65], TT-CNN [66, 67], & \multirow{3}{*}{3.1} \\  & & TR-CNN [68], BTT-CNN [69], TC-CNN [70, 71], HT-TT-CNN [72], & & \\  & & T-Net [73], TMTMT [74], PMT [75], CP-HOConv [76] & & \\ \cline{2-4}  & \multirow{3}{*}{**Network Compression**} & \multirow{3}{*}{Recurrent Neural Networks} & TT-RNN [77], BTT-RNN [69, 78], TR-RNN [79], HT-RNN [80], & \multirow{3}{*}{3.2} \\  & & CP/Tateer-RNN [84], TC-RNN [70, 71] Kronecker-RNN [81], & & \\  & & Tucker/CP-TT-RNN [82], HT-TT-RNN [72], KCP-RNN [83], & & \\  & & Conv-TT-LSTM [84] & & \\ \cline{2-4}  & \multirow{3}{*}{Transformer} & BTT-Transformer [85], MPO-Transformer [86], & & \\  & & Tuformer [87], Tucker-Bert [88], Hypoformer [89] & & \\ \cline{2-4}  & \multirow{3}{*}{Graph Neural Networks} & TGNNs [90], TGCN [91], DSTGNN [92] & & \\ \cline{2-4}  & & Restricted Boltzmann Machines & TT-RBM [93], TR-RBM [94], Tv-RBM [95], Mv-RBM [96] & & 3.5 \\ \hline \multirow{2}{*}{**Information Fusion**} & \multirow{2}{*}{Tensor Fusion Layers} & TFL [90, 97, 98], LMF [99], PFN [100] & & 4.1 \\ \cline{2-4}  & & Multimodal Pooling Layers & MUTAN [101], MCB [102], MLB [103] CIT [104] & & 4.2 \\ \hline \multirow{2}{*}{**Quantum Circuit**} & \multirow{2}{*}{Quantum Embedding} & Image-Emb [105], Language-Emb [106, 107, 108] & & 5.1 \\ \cline{2-4}  & \multirow{2}{*}{Quantum Data Processing} & Supervised MPS [105], Tree-TN [109], Uniform-MPS [108], LPS [110] & & 5.2 \\ \cline{2-4}  & & Quantum TNNs & ConvAC [107, 111], TLSM [112] & & 5.3 \\ \hline \hline \multirow{8}{*}{**Utility of TNNs**} & \multirow{3}{*}{Basic Tensor Operations} & Stable Training & Mixed Precision [113], Yu Initialization [114] & & 6.1 \\ \cline{2-4}  & \multirow{3}{*}{Basic Tensor Operations} & \multirow{3}{*}{Rank Selection} & PSTRN [115], TR-RL [116], CP-Bayes [117], & & 6.2 \\ \cline{1-1}  & & TT-Bayes [118], TT-ADMM [119], BMF [120, 121] & & 6.2 \\ \cline{2-4}  & \multirow{3}{*}{Hardware Speedup} & TIE [122], LTNN [123], TT-Engine [124], Fast CP-CNN [125] & & 6.3 \\ \cline{2-4}  & & Tensorly [126], TensorTools [127],Tensor Toolbox [128], & & \\ \cline{2-4}  & \multirow{3}{*}{Basic Tensor Operations} & TenDeC++ [129], OSTD [130], TensorD [131], TT-Toolbox [132], & & \\ \cline{2-4}  & & Trontoch [133], TronMPS [134], Tensor [135],T3F [136], & & 7.1 \\ \cline{2-4}  & \multirow{3}{*}{Deep Model Implementations} & Tensorly-Torch [126], TedNet [64] & & 7.2 \\ \cline{2-4}  & & Quantum Tensor Simulations & Yao [139], TensorNetwork [137], lambeq [140], ITensor [135], TeD-Q [141] & 7.3 \\ \hline \hline \end{tabular}
\end{table} TABLE I: An overview of TNNs and their utility. We first introduce the building of compact TNNs in different basic NN structures including CNNs, RNNs, Transformers, graph neural networks (GNNs) and RBMs in Section 3. Next, we explore the use of TNs in efficient information fusion methods based on tensor fusion and multimodal fusion in Section 4. Then, we discuss some applications involving TNs in quantum circuits and quantum TNNs in Section 5. Furthermore, we explain some training and implementation techniques for TNNs in Section 6. Finally, we introduce some general and powerful toolboxes for processing TNNs in Section 7.
compress the number of parameters by approximately 34,000 times while simultaneously outperforming naive LSTM.

(2) Information Fusion. In real-world data analysis cases, it is important to model higher-order interactions in data from multimodal sources to achieve better performance [8]. However, NNs are typically used to handle the inputs of one-mode vectors, so they lack sufficient expressive power to model such higher-order interactions [101]. To solve this problem, a promising approach is to embed TNs into NNs as efficient fusion units to process multimodal data with the help of the multiple-entry property [97, 98, 100]. Taking a visual question answering (VQA) task [142] as an example, multimodal Tucker fusion (MUTAN) [101] can learn high-level interactions between textual representations and visual representations via a Tucker-format framework. As a result, MUTAN has achieved state-of-the-art performance with an efficiently parameterized low-rank structure.

(3) Quantum Circuit Simulation. TNs can act as simulators and be bridges between classic NNs and quantum circuits. First, many studies have suggested implementing NNs on quantum circuits to accelerate their running speeds through the ultra-parallelism properties of quantum computation schemes [143, 144]. However, currently quantum computers are not sufficiently powerful for deploying NNs directly, which causes difficulty when verifying the possible performance of quantum neural networks (QNNs) [143]. Fortunately, TNs can be effective quantum simulators in electronic computers because of the equivalences between TNs and quantum circuits [145, 8]. In detail, the input qubits and unitary operation gates in quantum circuits can be viewed as tensors. Gate connections can also be viewed as tensor contractions in TN schemes [145]. By utilizing TNs to achieve quantum circuit simulation for NNs, a new era of QNNs exploration can be started before realistically powerful quantum computers are manufactured.

We call this family of approaches that connect TNs with NNs **tensorial neural networks (TNNs)**. To the best of our knowledge, this is the first comprehensive survey to bridge the connections among NNs, TNs, and Quantum Circuits. An overview of TNNs and their utility is shown in Table I.

The remaining sections of this survey are organized as follows. Section 2 provides the fundamentals of tensor notations, tensor diagrams, and TN formats. Section 3 discusses the use of TNs for building compact TNNs. Section 4 explores efficient information fusion processes using TNNs. Section 5 discusses some basic applications of TNs in quantum circuits and TNNs. Section 6 explains some training and implementation techniques for TNNs. Section 7 introduces general and powerful toolboxes that can be used to process TNNs.

## 2 Tensor Basis

### _Tensor Notations_

A tensor [146, 147], also known as a multiway array, can be viewed as a higher-order extension of a vector (i.e., a first-order tensor) or a matrix (i.e., a second-order tensor). Like the rows and columns in a matrix, an \(N\)th-order tensor \(\boldsymbol{\mathcal{X}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots\times I_{N}}\) has \(N\) modes (i.e., ways, orders, or indices) whose lengths (i.e., dimensions) are represented by \(I_{1}\) to \(I_{N}\), respectively. As shown in Table II, lowercase letters denote scalars, e.g., \(a\), boldface lowercase letters denote vectors, e.g., \(\mathbf{a}\), boldface capital letters denote matrices, e.g., \(\mathbf{A}\) and boldface Euler script letters denote higher-order tensors, e.g., \(\boldsymbol{\mathcal{A}}\). In this paper, we define a "tensor" with a wider range that includes scalars, vectors, and matrices.

### _Tensor Diagrams_

In this subsection, we introduce TN diagrams and their corresponding mathematical operations. TN diagrams were first developed by Roger Penrose [148] in the early 1970s and is now commonly used to describe quantum algorithms [8, 9] and machine learning algorithms [105, 61, 12]. In these diagrams, tensors are denoted graphically by nodes with edges [22]. TN diagrams are practical tools for the intuitive presentation and convenient representation of complex tensors. Therefore, tensor diagrams are widely used in the tensor field. As the data and weights in the deep learning field are all tensors, tensor diagrams are also promising for use as general network analysis tools in this area. An overview of the basic symbols of tensors is shown in Fig. 1.

#### 2.2.1 Tensor Nodes

A tensor is denoted as a node with edges, as illustrated in Fig. 1. The number of edges denotes the modes of a tensor, and a value on an edge represents the dimension of the corresponding mode. For example, a one-edge node denotes a vector \(\mathbf{a}\in\mathbb{R}^{I}\), a two-edge node denotes a matrix \(\mathbf{A}\in\mathbb{R}^{I\times J}\) and a three-edge node denotes a tensor \(\boldsymbol{\mathcal{A}}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}}\).

#### 2.2.2 Tensor Contraction

Tensor contraction means that two tensors are contracted into one tensor along their associated pairs of indices. As a result, the corresponding connected edges disappear while the dangling edges persist. Tensor contraction can be formulated as a tensor product:

\[\boldsymbol{\mathcal{C}}=\boldsymbol{\mathcal{A}}\times_{M+1,M+2, \ldots M+N}^{1,2,\ldots N}\boldsymbol{\mathcal{B}} \tag{1}\] \[=\sum_{i_{1},i_{2},\ldots i_{N}}\boldsymbol{\mathcal{A}}_{i_{1},i_{2},\ldots i_{N},\star}\quad\boldsymbol{\mathcal{B}}_{\star,i_{1},i_{2}, \ldots i_{N}}, \tag{2}\]

where \(\boldsymbol{\mathcal{A}}\in\mathbb{R}^{I_{1}\times\ldots I_{N}\times P_{1} \times\ldots P_{K}},\boldsymbol{\mathcal{B}}\in\mathbb{R}^{P_{K+1}\times \ldots P_{K+M}\times I_{1}\times\ldots I_{N}}\), and \(\boldsymbol{\mathcal{C}}\in\mathbb{R}^{P_{1}\times\ldots P_{K}\times P_{K+1} \cdots P_{K+M}}\). Fig. 1 also shows a diagram

\begin{table}
\begin{tabular}{c|c} \hline Symbol & Explanation \\ \hline \(a\) & scalar \\ \(\mathbf{a}\) & vector \\ \(\mathbf{A}\) & matrix \\ \(\boldsymbol{\mathcal{A}}\) & tensor \\ \(A\) & dimensionality \\ \(\oplus\) & convolution operation \\ \(\circ\) & outer product operation \\ \(<\cdot,\cdot>\) & inner product of two tensors \\ \(\lfloor\cdot\rfloor\) & quantum state bra vector(unit column complex vector) \\ \(\langle\cdot\rfloor\) & quantum state ket vector(unit row complex vector ) \\ \(\langle\cdot\lfloor\cdot\rfloor\rangle\) & inner product of two quantum state vectors \\ \hline \end{tabular}
\end{table} TABLE II: Tensor notations.

Fig. 1: Basic symbols for TN diagrams. For more basic knowledge about TNs, refer to [8] and [11].
of the matrix multiplication operation, which is the most classic tensor contraction situation. The equation representation is:

\[\mathbf{C}=\mathbf{A}\times_{2}^{1}\mathbf{B}. \tag{3}\]

Tensor contractions among multiple tensors (e.g., TNs) can be computed by sequentially performing tensor contractions between each pair of tensors. It is worth mentioning that the contracting sequence must be determined to achieve better calculation efficiency [149].

#### 2.2.3 Dummy Tensor

Recently, a newly designed dummy tensor was proposed by Hayashi et al. to represent convolution operations [61]. As depicted in Fig. 1, a node with star and arrow symbols denotes a dummy tensor. This operation is formulated as

\[\mathbf{y}_{j^{\prime}}=\sum_{j=0}^{\alpha-1}\sum_{k=0}^{\beta-1}\mathcal{P}_{ j,j^{\prime},k}\mathbf{a}_{j}\mathbf{b}_{k}, \tag{4}\]

where \(\mathbf{a}\in\mathbb{R}^{\alpha}\) denotes a vector that will be processed by a convolutional weight \(\mathbf{b}\in\mathbb{R}^{\beta}\), and \(\mathbf{y}\in\mathbb{R}^{\alpha^{\prime}}\) is an output. \(\mathcal{P}\in\left\{0,1\right\}^{\alpha\times\alpha^{\prime}\times\beta}\) is a binary tensor with elements defined as \(\mathcal{P}_{j,j^{\prime},k}=1\) if \(j=sj^{\prime}+k-p\) and \(0\) otherwise, where \(s\) and \(p\) represent the stride and padding size, respectively. Thus, \(\mathcal{P}\) can be applied to any two tensors to form a convolutional relationship.

#### 2.2.4 Hyperedge

As shown in Fig. 1, we illustrate the hyperedge that was also introduced by Hayashi et al. [61]. An example of a hyperedge with a size of \(R\) can be formulated as

\[\mathbf{y}_{ijk}=\sum_{l=1}^{R}\mathbf{A}_{il}\mathbf{B}_{jl}\mathbf{C}_{kl}, \tag{5}\]

where \(\mathbf{A}\in\mathbb{R}^{I\times R},\mathbf{B}\in\mathbb{R}^{J\times R}\) and \(\mathbf{C}\in\mathbb{R}^{K\times R}\) are three matrices. \(\mathbf{y}\in\mathbb{R}^{I\times J\times K}\) denotes the results of applying a hyperedge on \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\). A hyperedge node is simply equal to a tensor whose diagonal elements are 1. This tensor indicates the addition operation performed over several substructures (e.g., the matrices in Fig. 1). Hayashi et al. [61] showed that a tensor diagram can represent an arbitrary tensorial CNN (TCNN) by introducing dummy tensors and hyperedges.

#### 2.2.5 Tensor Unfolding

Tensor unfolding is an operation that virtually flattens a tensor into a high-dimensional but low-order tensor. Matricization is a special case of tensor unfolding. To be more specific, given an \(N\)th-order tensor \(\boldsymbol{\mathcal{A}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots\times I_{N}}\), its mode-\(n\) unfolding process yields a matrix \(\boldsymbol{A}_{(n)}\in\mathbb{R}^{I_{n}\times I_{1}I_{2}\ldots I_{n-1}I_{n+1} \ldots I_{N}}\). Such an operation can also be regarded as performing tensor contraction with a specifically designed tensor. A fourth-order tensor unfolding diagram is illustrated in Fig. 1.

### _Tensor Decomposition Formats_

The commonly used terminology "tensor decomposition" (TD) is equivalent to "tensor network" to some extent. Previously, TD was employed primarily in signal processing fields [150, 151]. TNs were originally utilized largely in the physics and quantum circuit fields [8, 148]. Traditional TD models, such as CP [13, 14, 15] and Tucker decomposition [16, 17], can be viewed as basic kinds of TNs. In the realm of signal processing, several powerful TNs architectures for quantum analysis have also been introduced. For instance, MPS decomposition [152] was defined as TT decomposition [21] and has tremendous success in several applications [12]. After years of collaboration and progress across different research fields, there is no significant distinction between these two terminologies. Therefore, TD and TNs are treated in a unified way in this paper. We briefly introduce some basic TDs by employing TN diagrams.

#### 2.3.1 Candecomp/parafafac

CP [13, 14, 15] factorizes a higher-order tensor into a sum of several rank-1 tensor components. For instance, given an \(N\)th-order

Fig. 2: Implementation of CP on a 5th-order tensor \(\mathcal{T}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}\times I_{4}\times I_{5}}\). We show the higher-order tensor in a matrix representation of size \(I_{1}\times I_{2}\), whose elements are 3rd-order tensors of size \(I_{2}\times I_{3}\times I_{4}\). The main idea of CP is to decompose \(\mathcal{T}\) into five factor vectors that encode the modes of \(\mathcal{T}\). Then, it is feasible to apply outer production on these factor vectors to recover \(\mathcal{T}\). Moreover, by regarding such an outer production as a component, CP with a rank of \(R\) implements summation on \(R\) components to further improve the expressive power of the tensor.
tensor \(\mathbf{\mathscr{X}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots I_{N}}\), each of its elements in the CP format form can be formulated as

\[\mathbf{\mathscr{X}}_{i_{1},i_{2},\ldots,i_{N}}\approx\sum_{r=1}^{R}\mathbf{\mathscr{S}}_ {r}\prod_{n=1}^{N}\mathbf{\mathsf{A}}_{i_{n},r}^{(n)}, \tag{6}\]

where \(R\) denotes the CP rank (defined as the smallest possible number of rank-1 tensors [150]), \(\mathbf{\mathscr{S}}\) denotes the diagonal core tensor (only the \(R\) nonzero elements on the superdiagonal) and \(\mathbf{\mathsf{A}}^{(n)}\in\mathbb{R}^{I_{n}\times R}\) denotes a series of factor matrices. The TN diagram for CP is illustrated in Fig. 3 (a). We also provide a detailed visualization of CP in Fig. 2 as an illustrative case of a TN.

When calculating a CP format, the first issue that arises is how to determine the number of rank-1 tensor components, i.e., the CP rank \(R\). Actually, this is an NP-hard problem [153]. Hence, in practice, a numerical value is usually assumed in advance (i.e., as a hyperparameter), to fit various CP-based models [150]. After that, the diagonal core tensor \(\mathbf{\mathscr{S}}\) and the factor matrices \(\mathbf{\mathsf{A}}^{(n)}\) can be directly solved by employing algorithmic iteration, which usually involves the alternating least-squares (ALS) method that was originally proposed in [13, 14].

#### 2.3.2 Tucker Decomposition

Tucker decomposition [16, 17] factorizes a higher-order tensor into a core tensor multiplied by a corresponding factor matrix along each mode. To be more specific, given an \(N\)th-order tensor \(\mathbf{\mathscr{X}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots I_{N}}\), the Tucker decomposition can be formulated in an elementwise manner as

\[\mathbf{\mathscr{X}}_{i_{1},i_{2},\ldots,i_{N}}\approx\sum_{r_{1},\ldots,r_{N}=1}^ {R_{1},\ldots,R_{N}}\mathbf{\mathscr{S}}_{r_{1},r_{2},\ldots,r_{N}}\prod_{n=1}^{N }\mathbf{\mathsf{A}}_{i_{n},r_{n}}^{(n)}, \tag{7}\]

where \(\{R_{1},R_{2},\ldots,R_{N}\}\) denotes a series of Tucker ranks, \(\mathbf{\mathscr{S}}\in\mathbb{R}^{R_{1}\times R_{2}\ldots R_{N}}\) denotes the core tensor and \(\mathbf{\mathsf{A}}^{(n)}\in\mathbb{R}^{I_{n}\times R_{n}}\) denotes a factor matrix. The TN diagram for Tucker decomposition is illustrated in Fig. 3 (b). Here, please note that compared with the CP rank, \(R_{1},R_{2},\ldots,R_{N}\) can take different numerical values.

Tucker decomposition is commonly used and can be degraded to CP by setting the core tensor \(\mathbf{\mathscr{G}}\) as a superdiagonal tensor whose diagonal elements are 1. In addition, Tucker decomposition lacks constraints on its factors, leading to the nonuniqueness of its decomposition results, which is typically undesirable for practical applications due to the lack of explainability. Consequently, orthogonal limitations are always imposed on the component matrices, yielding the well-known and classical higher-order singular value decomposition (HOSVD) algorithm [154].

#### 2.3.3 BTT Decomposition

CP and Tucker decomposition both decompose a tensor into a core tensor multiplied by a matrix along each mode, while CP imposes an additional superdiagonal constraint on the core tensor for the sake of simplifying the structural information of the core tensor. A more generalized decomposition method called BTT decomposition [18] has been proposed to make a tradeoff between the CP and Tucker methods by imposing a block diagonal constraint

Fig. 3: TN diagrams of some popular decompositions. (a) Diagrams of the CP format. It decomposes a tensor \(\mathbf{\mathscr{X}}\) into a sum of several rank-1 tensors \(\mathbf{\alpha}_{i,r}^{(1)}\circ\mathbf{\alpha}_{i,r}^{(2)}\circ\cdots\circ\mathbf{\alpha}_ {i,r}^{(N)}\). (b) Diagrams of Tucker decomposition. It decomposes a tensor \(\mathbf{\mathscr{X}}\) into a core tensor \(\mathbf{\mathscr{G}}\) multiplied by a matrix \(\mathbf{\mathsf{A}}^{(n)}\) along the \(n\)th mode. (c) Diagram of block term decomposition. It decomposes a tensor \(\mathbf{\mathscr{X}}\) into a sum of several Tucker decompositions (on the right) with low Tucker ranks. (d) Diagram of TT decomposition. It decomposes a tensor \(\mathbf{\mathscr{X}}\) into a linear multiplication of a set of 3rd-order core tensors \(\mathbf{\mathscr{G}}^{(2)}\cdots\mathbf{\mathscr{G}}^{(N-1)}\) and two matrices \(\mathbf{\mathscr{G}}^{(1)},\quad\mathbf{\mathscr{G}}^{(N)}\). (e) Diagram of TR decomposition. It decomposes a tensor \(\mathbf{\mathscr{X}}\) into a set of 3rd-order core tensors and contracts them into a ring structure. (f) Diagram of HT Decomposition. It represents a tensor \(\mathbf{\mathscr{X}}\) as a tree-like diagram. For more basic knowledge about TNs, refer to [8] and [11].
on Tucker's core tensor. The TN diagram for BTT decomposition is illustrated in Fig. 3 (c).

BTT decomposition aims to decompose a tensor into a sum of several Tucker decompositions with low Tucker ranks. Specifically, the BTT decomposition of a 4th-order tensor \(\mathbf{\mathfrak{X}}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}\times I_{4}}\) can be represented by 6 nodes with special contractions. Here, \(\mathbf{\mathfrak{S}}\in\mathbb{R}^{R_{C}\times R_{T}\times R_{T}\times R_{T} \times R_{T}}\) denotes the \(R_{C}\) core tensors of the Tucker decompositions, and each \(\mathbf{\mathcal{A}}^{(n)}\in\mathbb{R}^{R_{C}\times I_{u}\times R_{T}}\) denotes the \(R_{C}\) corresponding factor matrices of the Tucker decompositions. Moreover, each element of \(\mathbf{\mathfrak{X}}\) is computed as

\[\mathbf{\mathfrak{X}}_{i_{1},i_{2},i_{3},i_{4}}\approx\sum_{r_{C}=1} ^{R_{C}}\sum_{r_{1},r_{2},r_{3},r_{4}=1}^{R_{T},R_{T},R_{T},R_{T}}\] \[\mathbf{\mathfrak{S}}_{r_{C},r_{1},r_{2},r_{3},r_{4}}\mathbf{\mathcal{A}} ^{(1)}_{r_{C},i_{1},r_{1}}\mathbf{\mathcal{A}}^{(2)}_{r_{C},i_{2},r_{2}}\mathbf{ \mathcal{A}}^{(3)}_{r_{C},i_{3},r_{3}}\mathbf{\mathcal{A}}^{(4)}_{r_{C},i_{4},r_{4}}, \tag{8}\]

where \(R_{T}\) denotes the Tucker rank (which means that the Tucker rank equals \(\{R_{T},R_{T},R_{T},R_{T}\}\)) and \(R_{C}\) represents the CP rank. Together, they are called BT ranks.

The advantages of BTT decomposition mainly depend on its compatibility with the benefits of the both CP and Tucker methods. The reason for this is that when the Tucker rank is equal to 1, BTT decomposition degenerates to CP; when the CP rank equals 1, it degenerates to Tucker decomposition.

#### 2.3.4 TT Decomposition

TT decomposition [21, 22], also called MPS decomposition in quantum physics [152, 155], is derived purely from TNs. TT decomposition factorizes a higher-order tensor into a linear multiplication of a series of 3rd-order core tensors. For example, given an \(N\)th-order tensor \(\mathbf{\mathfrak{X}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots I_{N}}\), the TT decomposition can be formulated in an elementwise manner as

\[\mathbf{\mathfrak{X}}_{i_{1},i_{2},\ldots,i_{N}}\approx\sum_{r_{1},r_ {2},\ldots,r_{N-1}=1}^{R_{1},R_{2},\ldots,R_{N-1}}\] \[\mathbf{\mathfrak{S}}^{(1)}_{1,i_{1},r_{1}}\mathbf{\mathfrak{S}}^{(2)}_{ r_{1},i_{2},r_{2}}\mathbf{\mathfrak{S}}^{(3)}_{r_{2},i_{3},r_{3}}\cdots\mathbf{ \mathfrak{S}}^{(N)}_{r_{N-1},i_{N},1}, \tag{9}\]

where \(\{R_{1},R_{2},\ldots,R_{N-1}\}\) denote the TT ranks, \(\mathbf{\mathfrak{S}}^{(n)}\in\mathbb{R}^{R_{n-1}\times I_{n}\times R_{n}}\) denotes a 3rd-order core tensor and \(R_{0}=R_{N}=1\), which means that \(\mathbf{\mathfrak{S}}^{(1)}\) and \(\mathbf{\mathfrak{S}}^{(N)}\) are actually two matrices. The TN diagram for TT decomposition is illustrated in Fig. 3 (d).

TT decomposition can be computed easily by employing SVD recursively. In addition, as the simplest model among the available TN, TT decomposition is widely applied in the theory and practice of TNs [9]. Notably, Eq. (9) and Fig. 3 (d) have an MPS format. Some papers [77, 89, 93] have also used TT decomposition with an MPO [156] format. Given a \(2N\)-order tensor \(\mathbf{\mathfrak{X}}\in\mathbb{R}^{I_{1}\times J_{1}\times I_{2}\times J_{2} \ldots I_{N}\times J_{N}}\), its MPO decomposition can be mathematically expressed as

\[\mathbf{\mathfrak{X}}_{i_{1},j_{1},i_{2},j_{2},\ldots,i_{N},j_{N}} \approx\sum_{r_{1},r_{2},\ldots,r_{N-1}=1}^{R_{1},R_{2},\ldots,R_{N-1}}\] \[\mathbf{\mathfrak{S}}^{(1)}_{1,i_{1},j_{1},r_{1}}\mathbf{\mathfrak{S}}^{(2 )}_{r_{1},i_{2},j_{2},r_{2}}\mathbf{\mathfrak{S}}^{(3)}_{r_{2},i_{3},j_{3},r_{3}} \cdots\mathbf{\mathfrak{S}}^{(N)}_{r_{N-1},i_{N},j_{N},1}, \tag{10}\]

where \(\{R_{1},R_{2},\ldots,R_{N-1}\}\) denote the ranks, \(\mathbf{\mathfrak{S}}^{(n)}\in\mathbb{R}^{R_{n-1}\times I_{n}\times R_{n}}\) denotes a 4th-order core tensor and \(R_{0}=R_{N}=1\), which means that \(\mathbf{\mathfrak{S}}^{(1)}\) and \(\mathbf{\mathfrak{S}}^{(N)}\) are actually two 3rd-order core tensors.

#### 2.3.5 TR Decomposition

TT benefits from fast convergence. However, it suffers from its two endpoints, which hinder the representation ability and flexibility of TT-based models. Thus, to release the power of a linear architecture, researchers link its endpoints to produce a ring format named a TR [25]. The TR decomposition of a tensor \(\mathbf{\mathfrak{X}}\in\mathbb{R}^{I_{1}\times I_{2}\ldots I_{N}}\) can be formulated as

\[\mathbf{\mathfrak{X}}_{i_{1},i_{2},\ldots,i_{N}}\approx\sum_{r_{0},r_ {1},\ldots,r_{N-1}}^{R_{0},R_{1},\ldots,R_{N-1}}\] \[\mathbf{\mathfrak{S}}^{(1)}_{r_{0},i_{1},r_{1}}\mathbf{\mathfrak{S}}^{(2 )}_{r_{1},i_{2},r_{2}}\mathbf{\mathfrak{S}}^{(3)}_{r_{2},i_{3},r_{3}}\cdots\mathbf{ \mathfrak{S}}^{(N)}_{r_{N-1},i_{N},r_{0}}, \tag{11}\]

where \(\{R_{0},R_{1},\ldots,R_{N}\}\) denote the TR ranks, each node \(\mathbf{\mathfrak{S}}^{(n)}\in\mathbb{R}^{R_{n-1}\times I_{n}\times R_{n}}\) is a 3rd-order tensor and \(R_{0}=R_{N}\). Compared with TT decomposition, it is not necessary for TR decomposition to follow a strict order when multiplying its nodes. The TN diagram for TR decomposition is illustrated in Fig. 3 (e).

#### 2.3.6 HT Decomposition

HT decomposition [26] possesses a tree-like structure. In general, it is feasible to transfer a tensor \(\mathbf{\mathfrak{X}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{N}}\) to a binary tree with a root node associated with \(S_{set}=\{1,I_{2},\cdots,N\}\) and \(\mathbf{\mathfrak{X}}=\mathbf{\mathfrak{U}}_{S_{set}}\) as the root frame. \(S_{set1},S_{set2}\subseteq S_{set}\) is defined as the index set, which is associated with the left child node \(\mathbf{\mathfrak{U}}_{S_{set1}}\) and right child node \(\mathbf{\mathfrak{U}}_{S_{set2}}\). \(\mathbf{\mathfrak{U}}_{S_{set1}}\in\mathbb{R}^{R_{1}\times I_{\text{min}}(S_{set 1})\times\cdots\times I_{\text{max}}(S_{set1})}\) can also be recursively decomposed into its left child node \(\mathbf{\mathfrak{U}}_{D_{set1}}\) and right child node \(\mathbf{\mathfrak{U}}_{D_{set1}}\). The first three steps are as

\[\mathbf{\mathfrak{U}}_{S_{set}}\approx\mathbf{\mathfrak{S}}_{s}\times_{1}^{2} \mathbf{\mathfrak{U}}_{S_{set1}}\times_{1}^{2}\mathbf{\mathfrak{U}}_{S_{set2}}, \tag{12}\] \[\mathbf{\mathfrak{U}}_{S_{set1}}\approx\mathbf{\mathfrak{S}}_{s1}\times_{1}^ {2}\mathbf{\mathfrak{U}}_{D_{set1}}\times_{1}^{2}\mathbf{\mathfrak{U}}_{D_{set2}},\] (13) \[\mathbf{\mathfrak{U}}_{S_{set2}}\approx\mathbf{\mathfrak{S}}_{s2}\times_{1}^ {2}\mathbf{\mathfrak{U}}_{D_{set3}}\times_{1}^{2}\mathbf{\mathfrak{U}}_{D_{set4}}, \tag{14}\]

where \(\mathbf{\mathfrak{S}}_{s}\in\mathbb{R}^{R_{1}\times R_{2}}\), \(\mathbf{\mathfrak{S}}_{s1}\in\mathbb{R}^{R_{1}\times R_{3}\times R_{4}}\) and \(\mathbf{\mathfrak{S}}_{s1}\in\mathbb{R}^{R_{2}\times R_{5}\times R_{6}}\). This procedure can be performed recursively to obtain a tree-like structure. The TN diagram for HT decomposition is illustrated in Fig. 3 (f).

#### 2.3.7 PEPS Decomposition

A TN structure with different typologies and higher-dimensional connections can also be considered. PEPS decomposition [8, 27, 28], also known as tensor grid decomposition [157], is a high-dimensional TN that generalizes a TT. PEPS decomposition provides a natural structure that can capture more high-dimensional information. PEPS cores can be characterized as \(\mathbf{\mathfrak{S}}^{(m,n)}\in\mathbb{R}^{I_{mn}\times R_{imn}\times R_{mn} \times R_{mn}\times R_{dmn}}\). The mathematical formula [75] is

\[\mathbf{\mathfrak{X}}_{i_{1},i_{2},\ldots,i_{MN}}=\sum_
decomposition is illustrated in Fig. 3 (g). PEPS decomposition has a polynomial correlation decay with the separation distance. In contrast, MPS decomposition has an exponential correlation decay. This indicates that PEPS decomposition has a more powerful representation ability [8] because it strengthens the interactions between different tensor modes.

## 3 Network Compression with TNNs

DNNs have extraordinarily high spatial and temporal complexity levels, as deeply stacked layers contain large-scale matrix multiplications. As a result, DNNs usually require several days for training while occupying a large amount of memory for inference purposes. In addition, large weight redundancy has been proven to exist in DNNs [158], indicating the possibility of compressing DNNs while maintaining performance. Motivated by this, a wide range of compression techniques have been developed, including pruning [159, 160], quantization [161, 162], distillation [163, 164] and low-rank decomposition [79, 165, 166]. Among them, applying TNs to DNNs to construct TNNs can be a good choice since TNNs have excellent abilities to approximate the original weights with many fewer parameters [113]. In this direction, researchers have completed many studies, especially concerning the reconstruction of convolutional and fully connected layers through a variety of TD formats [61, 67, 69, 79]. With compact architectures, these TNNs can achieve improved performance with less redundancy. In this section, we introduce five common kinds of TNNs, i.e., TCNNs in Section 3.1, tensorial RNNs (TRNNs) in Section 3.2, tensorial Transformers in Section 3.3, tensorial GNN (TGNN) in Section 3.4, and tensorial RBMs in Section 3.5.

### _TCNNs_

CNNs have recently achieved much success. However, CNNs' enormous sizes cause weight redundancy and superfluous computations, affecting both their performance and efficiency. TD methods can be effective solutions to this problem. Commonly, CNNs represented with tensor formats are called TCNNs. Prior to introducing TCNNs, we formulate a vanilla CNN, shown in Fig. 4 (a), as

\[\mathfrak{Y}=\mathfrak{X}\oplus\mathfrak{C}+\mathbf{b}, \tag{17}\]

where \(\mathfrak{C}\in\mathbb{R}^{K\times K\times I\times O}\) denotes a convolutional weight, \(\mathfrak{X}\in\mathbb{R}^{I\times H\times W}\) denotes an input, \(\mathfrak{Y}\in\mathbb{R}^{O\times H^{\prime}\times W^{\prime}}\) denotes an output, \(\mathbf{b}\in\mathbb{R}^{O}\) represents a bias, and \(\otimes\) denotes a convolutional operator. \(K\) represents the kernel window size, \(I\) is an input channel, \(H\) and \(W\) denote the height and width of \(\mathfrak{X}\), \(O\) is an output channel, and \(H^{\prime}\) and \(W^{\prime}\) denote the height and width of \(\mathfrak{Y}\), respectively. TCNNs mainly focus on decomposing channels \(I\) and \(O\). In detail, the weight \(\mathfrak{C}\) is first reshaped to \(\tilde{\mathfrak{C}}\in\mathbb{R}^{K\times K\times I_{1}\times I_{2}\times \cdots I_{M}\times O_{1}\times O_{2}\times\cdots\otimes_{N}}\), where \(\prod_{k=1}^{M}I_{k}=I\) and \(\prod_{k=1}^{N}J_{k}=J\). Then, TCNNs can be derived by tensorizing the reshaped convolutional kernel \(\tilde{\mathfrak{C}}\).

To accelerate the CNN training and inference process, CP-CNN [61, 62, 63] is constructed by decomposing the convolutional weight into the CP format, as shown in Fig. 4 (d). CP-CNN only contains vectors as subcomponents, leading to an extremely compact structure and the highest compression ratio. As with CP-CNN, it is possible to implement additional TCNNs by applying tensor formats (as seen in the examples in Fig. 3) to the convolutional weight. Tucker decomposition, a widely used tensor format, is often applied to CNNs to form Tucker-CNNs [64, 65]. Different from simple Tucker formats, a BTT-CNN has a hyperedered \(R_{c}\), which can denote the summation of Tucker decompositions. Other BTT-CNNs [69] have also been proposed. Compared to Tucker CNNs, BTT-CNNs are much more powerful and usually derive better results [69]. Highly compact TT formats have also been introduced to CNNs to implement TT-CNNs [66]. Compared to TTs, TR formats are usually much more compact [68], and TR-CNNs [68] are much more powerful than TT-CNNs.

There are also some tensorial convolutional neural networks that decompose more than just the convolution cores. The tensorized network (T-Net) [73] treats the whole network as a one-layer architecture and then decomposes it. As a result, the T-Net achieves better results with a lighter structure. CP-higher-order convolution (CP-HOConv) [76] utilizes the CP format to handle tasks with higher-order data, e.g., spatiotemporal emotion estimation. For multitask missions, Yang et al. [74] proposed the Tensor Train multitask (TTMT) and Tucker multitask (TMT) models using TT and Tucker formats, respectively, to alleviate the negative transfer problem in a hard sharing architecture and reduce the parameter volume in a soft structure. A PEPS-like concatenated TN layer [75] for multitask missions was also proposed. Unlike the TMT and TMT models, which suffer from the negative transfer problem due to their hard sharing architectures, the PEPS structure only contains a soft sharing layer, thereby achieving better performance.

Fig. 4: Correspondence between TN diagrams and convolutional procedures. In each subfigure, the left part is a TN diagram, and the right part is the associated commonly used feature representation.
### _TrNNs_

RNNs, such as the vanilla RNN and LSTM, have achieved promising performance on sequential data. However, when dealing with high-dimensional input data (e.g., video and text data), the input-to-hidden and hidden-to-hidden transformations in RNNs will result in high memory usage rates and computational costs. To solve this problem, low-rank TD is efficient for compressing the transformation process in practice. First, we formulate an RNN as

\[\mathbf{h}^{(t+1)}=\phi(\mathbf{W}\mathbf{x}^{(t)}+\mathbf{U}\mathbf{h}^{(t)}+ \mathbf{b}), \tag{18}\]

where \(\mathbf{h}^{(t)}\in\mathbb{R}^{O}\) and \(\mathbf{x}^{(t)}\in\mathbb{R}^{I}\) denote the hidden state and input feature at time \(t\), respectively, \(\mathbf{W}\in\mathbb{R}^{O\times I}\) is the input-to-hidden matrix, \(\mathbf{U}\in\mathbb{R}^{O\times O}\) represents the hidden-to-hidden matrix, and \(\mathbf{b}\in\mathbb{R}^{O}\) is a bias. \(\phi(\cdot)\) indicates a series of operations that form RNN variants, including the vanilla RNN and LSTM [167]. Eq. (18) can also be reformulated in a concatenation form that is widely used in TD:

\[\mathbf{h}^{(t+1)}=\phi([\mathbf{W},\mathbf{U}][\mathbf{x}^{(t)},\mathbf{h}^ {(t)}]+\mathbf{b}), \tag{19}\]

where \([\mathbf{W},\mathbf{U}]\in\mathbb{R}^{O\times(I+O)}\) and \([\mathbf{x}^{(t)},\mathbf{h}^{(t)}]\in\mathbb{R}^{(I+O)}\) denote the concatenation of \(\mathbf{W},\mathbf{U}\) and \(\mathbf{x}^{(t)},\mathbf{h}^{(t)}\), respectively. As shown in Fig. 5, there are usually two ways to decompose RNNs: (a) only tensorizing \(\mathbf{W}\), which is often the largest component in an RNN, and (b) tensorizing \([\mathbf{W},\mathbf{U}]\) for extreme compression. Note that since \(\mathbf{U}\) is usually smaller than \(\mathbf{W}\), no works decompose \(\mathbf{U}\) only. The process of implementing a TRNN is the same as that used to implement a TCNN, namely, reshaping the weights into higher-order formulations and replacing them with tensor formats.

The most direct and simple compression method is to solely decompose the enormous input-to-hidden matrix \(\mathbf{W}\). The CP-RNN and Tucker-RNN [64] can be directly constructed with the CP and Tucker formats, respectively. With an extremely compact low-rank structure, the CP-RNN can always derive the smallest size in comparison with other tensor formats. The TT-RNN [77] implements the TT format on an RNN to obtain a high parameter compression ratio. However, the TT-RNN suffers from a linear structure with two smaller endpoints, which hinders the representation ability and flexibility of TT-based models. To release the power of a linear architecture, TRs were proposed to link the endpoints to create a ring format [25]. TR-An RNN [79] with a TR was formed to achieve a much more compact network. BTT-RNN [69, 78] was constructed on the generalized TD approach: BTT decomposition [19]. BTT-RNN can automatically learn interparameter correlations to implicitly prune redundant dense connections and simultaneously achieve better performance.

Moreover, studies are utilizing TD to compress an RNN's two transformation layers, and some have even developed decomposition methods that are suitable for both RNNs and CNNs. TT-GRU [82] and the HT-RNN [80] decompose \([\mathbf{W},\mathbf{U}]\) to attain a higher compression ratio. Specifically, TT-GRU [82] applies a TT for decomposition, and the HT-RNN [80] adopts HT decomposition. Unlike prior works that decomposed hidden matrices, Conv-TT-LSTM [84] utilizes the idea of a TT to represent convolutional operations. As shown in Fig. 5, through a TT-like convolution, Conv-TT-LSTM can replace convolutional LSTM with fewer parameters while achieving good results on action benchmarks. For the adaptation of both CNNs and RNNs, a hybrid TD (termed HT-TT) method that combines HT and TT decomposition [72] was adopted to compress both the CNN and RNN \([\mathbf{W},\mathbf{U}]\) matrices. In addition, the tensor contraction layer (TC-Layer) [71] was designed to replace the fully connected layer and therefore can be utilized as the last layer of a CNN and the hidden layers in RNNs. Interestingly, TC-Layer is a special case of a TT-based layer obtained by setting the ranks to 1.

### _Tensorial Transformers_

Transformers [168, 46] are well known for processing sequence data. Compared with CNNs and RNNs, Transformers can be stacked into large-scale sizes to achieve significant performance [47]. However, Transformers are still redundant, similar to classic DNNs, which can be made smaller and more efficient [88]. Therefore, TD, as a flexible compression tool, can be explored to reduce the numbers of parameters in Transformers [85, 86, 87].

Classic Transformers mainly consist of self-attention (SA) and feedforward Networks (FFNs). SA processes the given query matrix \(\mathbf{Q}\), key matrix \(\mathbf{K}\) and value matrix \(\mathbf{V}\) with parameters \(\mathbf{W}^{Q},\mathbf{W}^{K},\mathbf{W}^{V},\mathbf{W}^{O}\). More generally, SA is separated into \(n\) heads: \(\{\mathbf{W}_{i}^{Q}\}^{n},\{\mathbf{W}_{i}^{K}\}^{n},\{\mathbf{W}_{i}^{V}\}^{n}, \{\mathbf{W}_{i}^{O}\}^{n}\). Each head can be calculated as

\[\mathrm{Att}_{i}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left( \frac{\mathbf{Q}\mathbf{W}_{i}^{Q}\mathbf{W}_{i}^{K}{}^{T}\mathbf{K}^{T}}{ \sqrt{d}}\right)\mathbf{V}\mathbf{W}_{i}^{V}{\mathbf{W}_{i}^{O}}^{T}. \tag{20}\]

Then, \(\mathrm{SA}((\mathbf{Q},\mathbf{K},\mathbf{V}))=\sum_{i=1}^{n}\mathrm{Att}_{i} (\mathbf{Q},\mathbf{K},\mathbf{V})\). Another important component, the FFN, is formulated as

\[\mathrm{FFN}(\mathbf{X})=\mathrm{ReLU}(\mathbf{X}\mathbf{W}^{in}+\mathbf{b}^{ in})\mathbf{W}^{out}+\mathbf{b}^{out}, \tag{21}\]

where \(\mathbf{X}\) is the input, \(\mathbf{b}^{in}\) and \(\mathbf{b}^{out}\) are biases, and \(\mathbf{W}^{in}\) and \(\mathbf{W}^{out}\) are weights. Apparently, the number of parameters in a Transformer is mainly based on its linear transformation matrices, i.e., \(\mathbf{W}^{Q},\mathbf{W}^{K},\mathbf{W}^{V},\mathbf{W}^{O}\), \(\mathbf{W}^{in}\) and \(\mathbf{W}^{out}\).

Therefore, most compression studies focus on eliminating the parameters of these matrices. For instance, the MPO structure was proposed to decompose each matrix in a Transformer [86], generating central tensors (containing the core information) and small auxiliary tensors. A tuning strategy was further adopted to continue training the auxiliary tensors to achieve a performance

Fig. 5: TR LSTM. It is effective at reducing the parameters of an LSTM model by replacing the input-to-hidden transformation weights with TR decomposition.
improvement while freezing the weight of the central tensor to retain the main information of the original matrix. Moreover, observing that a low-rank MPO structure can cause a severe performance drop, Hypoformer [89] was proposed based on hybrid TT decomposition; this approach concatenates a dense matrix part with a low-rank MPO part. Hypoformer retains the full-rank property while reducing the required numbers of operations and parameters to compress and accelerate the base Transformer. In addition, by concatenating all matrices into one larger tensor, Tucker-Bert [88] decomposes the concatenated tensor with Tucker decomposition to greatly reduce the number of parameters, leading to extreme compression and maintaining comparably good results. Interestingly, Tuformer [87] generalizes MHSA into the Tucker form, thus containing more expressive power and achieving better results, as shown in Fig. 6.

### _Tgnns_

GNNs have achieved groundbreaking performances across a range of applications and domains [169]. One classic GNN layer consists of an aggregation function for aggregating the neighbor node information and an update function for updating the current node information. For example, the processing step for node \(v\) in the \(k\)-th layer of a GNN can be formulated as

\[\begin{split}\mathbf{a}_{v}^{(k)}&\leftarrow \mathrm{Aggregate}_{(k)}\left(\left\{\mathbf{h}_{u}^{(k-1)},\forall u\in \mathcal{N}(v)\right\}\right),\\ \mathbf{h}_{v}^{(k)}&\leftarrow\mathrm{Update}_{(k) }\left(\mathbf{h}_{v}^{(k-1)},\mathbf{a}_{v}^{(k)}\right),\end{split} \tag{22}\]

where \(\mathbf{a}_{v}^{(k)}\) is an aggregated embedding vector, \(\mathbf{h}_{v}^{(k-1)}\) is a node embedding vector, and \(\mathcal{N}(v)\) is a neighbor node set. A typical choice for the update function is a simple-layer perceptron, and simple summation/maximization is always chosen as the aggregation function. Classic GNNs suffer from low model expressivity since high-order nonlinear information among nodes is missed [90]. Because of the merits of the tradeoff between expressivity and computing efficiency, the usage of TGNNs for graph data processing is quite beneficial.

To efficiently parameterize permutation-invariant multilinear maps for modeling the interactions among neighbors in an undirected graph structure, a TGNN [90] makes use of a symmetric CP layer as its node aggregation function. It has been demonstrated that a TGNN has a strong capacity to represent any multilinear polynomial that is permutation-invariant, including the sum and mean pooling functions. Compared to undirected graph processing, TGNNs are more naturally suited for high-order graph structures, such as knowledge graphs. Traditional relational graph convolutional networks neglect the trilinear interaction relations in knowledge graphs and additively combine the information possessed by entities. The TGCN [91] was proposed by using a low-rank Tucker layer as the aggregation function to improve the efficiency and computational space requirement of multilinear modeling. TGNNs are also appropriate for high-order correlation modeling in dynamic spatial-temporal graph processing situations. For example, The DSTGNN [92] applies learnable TTG and STG modules to find dynamic time relations and spatial relations, respectively. Then, the DSTGNN explores the dynamic entangled correlations between the STG and TTG modules via a PEPS layer, which reduces the number of DSTGNN parameters.

### _Tensorial RBMs_

RBMs [42] are generative stochastic NNs that can learn a probability distribution from an input set. A standard RBM consists of one visible factor \(\mathbf{v}\in\mathbb{R}^{M}\) and one hidden factor \(\mathbf{h}\in\mathbb{R}^{N}\) and assigns the following energy function for a joint vector \(\{\mathbf{v},\mathbf{u}\}\) as

\[E(\mathbf{v},\mathbf{h})=-\mathbf{v}^{T}\mathbf{w}\mathbf{h}-\mathbf{v}^{T} \mathbf{b}-\mathbf{c}^{T}\mathbf{h}, \tag{23}\]

where \(\mathbf{b}\in\mathbb{R}^{M}\) and \(\mathbf{c}\in\mathbb{R}^{N}\) are the biases of the visible layer and hidden layer, respectively, and \(\mathbf{W}\in\mathbb{R}^{M\times N}\) is the mapping weight matrix. The probability distribution of the joint vector \(\{\mathbf{v},\mathbf{u}\}\) can be defined as

\[P(v,h)=\frac{1}{Z}e^{-E(\mathbf{v},\mathbf{h})}, \tag{24}\]

where \(Z\) is a partition function. Then, some loss function can be defined via the learnable distribution formulation to optimize the parameters. The RBM parameters appear to be based mostly on the mapping weight matrix.

As a result, the majority of compression research concentrates on reducing the number of weight matrix variables. For instance, Tv-RBM [95] explores the use of an RBM for higher-order inputs. In this model, the weight matrix is transformed into a CP layer structure, where each visible layer is represented as a tensor while each hidden layer is still a vector. In another higher-order RBM, namely, Mv-RBM [96], its visible and hidden layers are all represented as matrices, and its weights are represented as a TC layer [71]. MPO-RBM [170] and TT-RBM [93] represent the weight matrix with a TT layer to greatly compress the number of required parameters. Moreover, TR-RBM [94] performs TR decomposition on its RBM, where the visible and hidden layers of TR-RBM are all generalized to tensors.

**Remark.** Compact TNNs have demonstrated the potential to achieve extremely high compression ratios while preserving their

Fig. 6: Tensor diagrams for SA modules [86]. (a) It is feasible to represent a classic multihead SA (MHSA) mechanism in a tensor diagram. MHSA can be treated as a special case of tunable-head self-attention (THSA) by setting \(\mathbf{C}=\mathbf{I}_{H}\otimes(\mathbf{1}_{D}^{\top}\mathbf{1}_{D})\). (b) The THSA) of the Tuformer can be a more generalized version of SA through a trainable matrix \(\mathbf{C}\). (c) THSA has a design space formulated as \(\mathbf{C}=\mathbf{C}_{1}\otimes(C_{2}^{\top}C_{3})\), which is the direct generalized form of MHSA.
model performance. However, their computational acceleration rates are not very significant compared with their compression ratios, which is mainly due to the contraction operations. This therefore calls for further research to improve the employed contraction strategies, since unoptimized contraction strategies can result in unsatisfactory running memory consumption.

## 4 Information Fusion via TNNs

In real-world data analyses, the collected data can be derived from multiple sources; e.g., vision, sound, and text sources can be contained in video data [142]. For example, in the VQA task, the key point lies in effectively modeling the interactions between the two modalities, i.e., text and image information. When processing such data, it is infeasible to consider diverse sources in the same form. Therefore, it is desirable to mix this information through multiple entrances to address multimodal sources in special building structures. Such methods with entrances are called information fusion approaches. Feature-level fusion [171] and decision-level fusion [172] are popular methods that are used in the early stage. However, these methods are simple linear methods and do not allow intramodality dynamics to be efficiently modeled. To solve this problem, TNNs are utilized in fusion tasks for modeling intramodality dynamics based on the natural multilinear property. In addition, TNNs are capable of processing higher-order data, which is a widely used ability. In conclusion, TNs provide effective frameworks for tensor operations, and it is natural and meaningful to express and generalize the information fusion modules (such as attention modules and vector concatenation modules) encountered in deep learning through TNs. Therefore, many studies adopt TNNs to capture the higher-order interactions among data or parameters. In this section, we introduce two main series of TNN structures for information fusion: the tensor fusion layer in Section 4.1 and multimodal pooling in Section 4.2.

### _Tensor Fusion Layer-Based Methods_

Multimodal sentiment analysis is a task containing three communicative modalities, i.e., the textual modality, visual modality, and acoustic modality [97]. Addressing multimodal sentiment analysis, Zadeh et al. [97] proposed novel TNNs with deep information fusion layers named tensor fusion layers (TFLs), which can easily learn intramodality dynamics and intermodality dynamics and are able to aggregate multimodal interactions, thereby efficiently fusing the three communicative modalities. Specifically, a TFL first takes embedded feature vectors \(\mathbf{z_{t}}\), \(\mathbf{z_{v}}\) and \(\mathbf{z_{a}}\) derived by embedding networks rather than the original three data types. Then, the TFL concatenates a scalar \(1\) with each embedded feature vector:

\[\mathbf{z}_{t}^{{}^{\prime}}=\left[\begin{array}{c}\mathbf{z_{t}}\\ 1\end{array}\right]\mathbf{z}_{v}^{{}^{\prime}}=\left[\begin{array}{c} \mathbf{z_{v}}\\ 1\end{array}\right]\mathbf{z}_{a}^{{}^{\prime}}=\left[\begin{array}{c} \mathbf{z_{a}}\\ 1\end{array}\right]. \tag{25}\]

Then, as shown in Fig. 7, the TFL obtains a feature tensor \(\mathsf{\Sigma}\) by calculating the outer product among the three concatenated vectors:

\[\mathsf{\Sigma}=\mathbf{z}_{t}^{{}^{\prime}}\circ\mathbf{z}_{v}^{{}^{\prime}} \circ\mathbf{z}_{a}^{{}^{\prime}}=\left[\begin{array}{c}\mathbf{z_{t}}\\ 1\end{array}\right]\circ\left[\begin{array}{c}\mathbf{z_{v}}\\ 1\end{array}\right]\circ\left[\begin{array}{c}\mathbf{z_{u}}\\ 1\end{array}\right]. \tag{26}\]

Finally, the TFL processes the feature tensor \(\mathsf{\Sigma}\) to obtain a prediction \(\mathbf{y}\) via a two-layer fully connected NN. Compared to direct concatenation-based fusion, which only considers unimodal interactions [97], the TFL benefits from capturing both unimodal interactions and multimodal interactions.

Despite its success, the TFL suffers from exponential increases in its computational complexity and number of parameters when the number of modalities increases. For example, in a multimodal sentiment analysis case [97], the feature tensor \(\mathsf{\Sigma}\in\mathbb{R}^{129\times 33\times 33}\) and the hidden vector \(\mathbf{h}\in\mathbb{R}^{128}\) can result in \(17,981,568\) parameters to be optimized. To address these excessive parameters, low-rank multimodal fusion (LMF) [99] adopts a special BTT layer to overcome the massive computational cost and overfitting risks of the TFL. For a general situation with \(n\) modalities, the feature tensor \(\mathsf{\Sigma}=\circ_{m=1}^{M}z_{m}^{{}^{\prime}}\) can be processed. The hidden vector \(\mathbf{h}\) can be computed as follows:

\[\mathbf{h}=ReLU\left(\mathbf{h}\mathbf{e}(\mathbf{W}_{1}z_{1}^{{}^{\prime}}, \mathbf{W}_{2}z_{2}^{{}^{\prime}},\cdots\mathbf{W}_{M}z_{M}^{{}^{\prime}},I)+ \mathbf{b}\right),\]

where \(\mathbf{W}_{i}\in\mathbb{R}^{d_{i}\times d_{h}}\) is the weight matrix and \(I\in\mathbb{R}^{d_{h}\times d_{h}}\) is an identity matrix. LMF reduces the computational complexity of the TFL from \(O\left(\prod_{m=1}^{M}d_{m}\right)\) to \(O\left(d_{h}\times\sum_{m=1}^{M}d_{m}\right)\).

Although LMF and the TFL achieve better fusion results than other methods, they restrict the order of interactions, causing higher-order interactions to lack information. A PTP [100] block has been proposed to tackle this problem. The whole procedure and TN diagram of PTP are shown in Fig. 8 and Fig. 9, respectively.

PTP first merges all feature vectors \(\left\{\mathbf{z}_{m}\right\}_{m=1}^{M}\) into a long feature vector

\[\mathbf{z}_{12\cdots M}^{\top}=\left[1,\mathbf{z}_{1}^{\top},\mathbf{z}_{2}^ {\top},\cdots,\mathbf{z}_{M}^{\top}\right]. \tag{27}\]

The polynomial feature tensor of degree \(P\) is represented as

\[\mathsf{\Sigma}^{P}=\mathbf{z}_{12\ldots M}\circ\mathbf{z}_{12\ldots M}\circ \cdots\circ\mathbf{z}_{12\cdots M}. \tag{28}\]

PTP [100] then adopts a tensorial layer (e.g., a CP layer) to process the polynomial feature tensor \(\mathsf{\Sigma}^{P}\). The CP layer is represented as

\[\mathbf{h}=\mathbf{h}\mathbf{e}(\mathbf{W}_{1}\mathbf{z}_{12\ldots M},\cdots\mathbf{ W}_{P}\mathbf{z}_{12\ldots M},\mathbf{\Lambda}) \tag{29}\]

where \(\mathbf{W}_{i}\in\mathbb{R}^{d_{i}\times d_{h}}\) is the weight matrix and \(\mathbf{\Lambda}\in\mathbb{R}^{d_{h}\times d_{h}}\) is a learnable diagonal matrix. The structure of PTP is also equivalent to that of a deep polynomial NN [173]. PTP models all nonlinear high-order interactions. For multimodal time series data, one approach uses a "window" to characterize local correlations and stack the PTP blocks in multiple layers. Such a model is called a hierarchical polynomial fusion network (HPFN) [100]. The HPFN can recursively process local temporal-modality patterns to achieve a better information fusion effect.

The structure of a single-layer PTP block is similar to that of a shallow convolutional arithmetic circuit (ConvAC) network [107] (see Section 5.3). The only difference between ConvAC and PTP is that the standard ConvAC network processes quantum location features, whereas PTP processes the temporal-modality patterns and polynomial concatenated multimodal features. The

Fig. 7: Illustration of the tensor fusion process in Eq. (26). Different from a TN diagram, each circle corresponds to a value.
HPFN is nearly equivalent to a deeper ConvAC network, and its great expressive power might be implied by their connection. The recursive relationships in deep polynomial NNs have also been found and implemented so that polynomial inputs can be efficiently computed via a hierarchical NN [100]. Chrysos et al. [173] also discovered similar results.

### _Multimodal Pooling-Based Methods_

Another group of information fusion methods originated from VQA tasks [142]. In VQA tasks, the most important aspect is to parameterize bilinear the interactions between visual and textual representations. To address this aspect, some tensor fusion methods have been discovered in this area. Multimodal compact bilinear pooling (MCB) [102] is a well-known fusion method for VQA tasks and can be regarded as a special Tucker decomposition-based NN. MCB tries to optimize the simple bilinear fusion operation

\[\mathbf{z}=\mathbf{W}[\mathbf{v}\circ\mathbf{q}], \tag{30}\]

where \(\mathbf{v}\) and \(\mathbf{q}\) are input vectors with different modalities and \(\mathbf{W}\) is a learnable weight matrix. Moreover, MCB optimizes the computational cost of the outer product operation based on the property of the count sketch projection function.

Multimodal low-rank bilinear pooling (MLB) [103] adopts a CP layer in a data fusion step that can be formulated as follows:

\[\mathbf{z}=\mathbf{1}^{T}\left(\mathbf{W}_{v}\mathbf{v}\circ\mathbf{W}_{q} \mathbf{q}\right), \tag{31}\]

where \(\mathbf{W}_{q}\) and \(\mathbf{W}_{v}\) are prepossessing weight matrices for inputs \(\mathbf{q}\) and \(\mathbf{v}\), respectively and \(\mathbf{1}\) is a vector in which all values are 1. The structure of the MLB method is a special case of LMF (see Sec. 4.1). MLB fusion methods can also be regarded as simple product pooling when the number of modalities is equal to two.

MUTAN [101] is a generalization of MCB and MLB. MUTAN adopts a Tucker layer to learn the bilinear interactions between visual and textual features:

\[\mathbf{z} =\left(\left(\boldsymbol{\mathcal{T}}_{c}\times_{1}^{1}\left( \mathbf{q}^{\top}\mathbf{W}_{q}\right)\right)\times_{2}^{1}\left(\mathbf{v}^{ \top}\mathbf{W}_{v}\right)\right)\times_{3}^{1}\mathbf{W}_{o},\] \[\mathbf{z} =\left(\boldsymbol{\mathcal{T}}_{c}\times_{1}^{1}\tilde{\mathbf{ q}}\right)\times_{2}^{1}\tilde{\mathbf{v}}, \tag{32}\]

where \(\tilde{\mathbf{q}}=\tanh\left(\mathbf{q}^{\top}\mathbf{W}_{q}\right)\) and \(\tilde{\mathbf{v}}=\tanh\left(\mathbf{v}^{\top}\mathbf{W}_{v}\right)\), \(\mathcal{T}_{c}\) is the fusion weight tensor, and \(\mathbf{W}_{o}\) is the output processing weight matrix. Moreover, MUTAN [101] adopts a low rank for the fusion weight tensor \(\boldsymbol{\mathcal{T}}_{c}\), as follows:

\[\boldsymbol{\mathcal{T}}_{c}[:,:,k]=\sum_{r=1}^{R}\mathbf{m}_{r}^{k}\circ \mathbf{n}_{r}^{k\top}, \tag{33}\]

where \(\mathbf{m}_{r}^{k}\) and \(\mathbf{n}_{r}^{k\top}\) are weight vectors and \(R\) is the number of ranks. MUTAN can represent comprehensive bilinear interactions while maintaining a reasonable model size by factorizing the interaction tensors into interpretable elements.

Furthermore, compact trilinear interaction (CTI) [104] was proposed to use an attention-like structure. Instead of presenting the given data as a single vector, this method represents every modality as a matrix \(A\in\mathbb{R}^{n_{1}\times d_{a}}\), where \(d_{a}\) corresponds to the feature dimension and \(n_{1}\) denotes the number of states. CTI simultaneously learns high-level trilinear joint representations in VQA tasks and overcomes both the computational complexity and memory issues in trilinear interaction learning [104].

**Remark.** When fusing information in many multimodal tasks, TNNs can achieve promising results with natural multilinear and compact frameworks. However, the high-order outer product operation used in TNNs may cause unexpected computational complexity increases and even unstable numerical properties. Therefore, it is important to consider an efficient algorithm for reducing memory consumption and apply a feasible initialization algorithm (e.g., [114]) to achieve good stability.

## 5 Quantum Circuit Simulation with TNNs

In the past few years, the development of quantum computing theory has attracted much attention [174, 175, 176]. Quantum systems have superiority in terms of parallelism over classic electronic computers [177], so they can achieve algorithms with lower time complexity. For example, Shor's algorithm [178] based on quantum systems is theoretically exponentially faster than the classic prime number decomposition algorithm. Quantum circuits are computational hardware implements of quantum systems, and they theoretically correspond to TNs and TNNs [179, 180]. Quantum states are mathematical entities of quantum systems and are consistent with higher-order tensors with some constraints [8]. Therefore, TNNs can be used as simulators in classic computers to model realistic quantum circuits [8, 145]. Taking advantage of the ultrahigh parallelism of quantum computing, some special TNNs can be implemented on small, near-term quantum devices [180]. Quantum circuit simulation on TNNs mainly focuses on the roles of TNs as bridges between classic NNs and QNNs rather than the more general TN-based quantum circuit simulation paradigm. Please refer to other papers [8, 30, 145] if readers are interested in general circuit simulation via TNs. In this section, we use the term "classic data" to denote the data in classic electronic computers. We introduce methods for mapping classic data to quantum states through TNs in Section 5.1, then introduce basic supervised and unsupervised processing methods for the mapped
quantum states in Section 5.2, and finally introduce the famous quantum TNN model, i.e., ConvAC in Section 5.3.

### _Quantum State Embedding for Classic Data_

To process machine learning tasks in a quantum system, the input data should be converted into a linear combination of some quantum states as an orthogonal basis:

\[\ket{\psi}=\sum_{d_{1},d_{N}=1}^{M}\mathcal{A}_{d_{1}\ldots d_{N}} \ket{\psi_{d_{1}}}\circ\cdots\circ\ket{\psi_{d_{N}}},\] \[s.t\quad\sum_{d_{1}\ldots d_{N}=1}^{M}\mathcal{A}_{d_{1}\ldots d _{N}}^{2}=1,\quad\mathcal{A}_{d_{1}\ldots d_{N}}\geq 0, \tag{34}\]

where \(\ket{\cdot}\) is the Dirac notation of a vector with complex values [181], and \(\circ\) denotes the outer product operation. The tensor \(\mathcal{A}\) is the combination coefficient tensor and is always represented and analyzed via a low-rank TN [8]. To embed classic data into a quantum state for adapting quantum systems, Stoudenmire and Schwab [105] proposed a quantum state mapping function \(\phi^{i}(x_{i})\) for the \(i\)-th pixel \(x_{i}\) in a grayscale image as

\[\phi^{i}(x_{i})=[\cos(\frac{\pi}{2}x_{i}),\sin(\frac{\pi}{2}x_{i})]. \tag{35}\]

The values of pixels are transformed into the range from 0.0 to 1.0 via the mapping function. Furthermore, a full grayscale image \(\mathbf{x}\) can be represented as outer products of the mapped quantum states of each pixel:

\[\Phi^{1,2,\ldots N}(\mathbf{x})=\phi^{1}(x_{1})\circ\phi^{2}(x_{2})\circ \cdots\phi^{N}(x_{N}), \tag{36}\]

where \(\Phi^{1,2,\ldots N}(\mathbf{x})\in\mathbb{R}^{\overbrace{2\times 2\cdots \times 2}^{N}}\). Through Eq. (36), it is feasible to associate realistic images with real quantum systems.

For a natural language document, the \(i\)-th word \(\ket{x_{i}}\) can also be represented as the sum of orthogonal quantum state bases \(\ket{\phi_{h_{i}}}(h_{i}=1,\ldots,M)\)[106, 107, 108, 182] corresponding to a specific semantic meaning \(M\):

\[\ket{x_{i}}=\sum_{h_{i}=1}^{M}\alpha_{i,h_{i}}\ket{\phi_{h_{i}}},\] \[s.t\quad\sum_{h_{i}=1}^{M}\alpha_{i,h_{i}}^{2}=1,\quad\alpha_{i, h_{i}}\geq 0, \tag{37}\]

where \(\alpha_{i,h_{i}}\) is the associated combination coefficient for each semantic meaning. The constraint of \(\alpha_{i}\) is and. After completing data mapping, the embedded quantum data can be processed by TNNs on a realistic quantum circuit, as shown in Fig. 10. The loss functions of TNNs can also be defined through the properties of quantum circuits. Such a procedure can be simulated on classic electronic computers via TNs and can be theoretically efficiently implemented on realistic quantum systems.

### _Embedded Quantum Data Processing_

Two series of learning methods are important and have potential for designing and optimizing TNNs, which can be implemented on realistic quantum circuits. One involves applying the density matrix renormalization group (DMRG) algorithm [8, 184] to train supervised models. The other adopts the ideas of the Born machine [185] to learn data distributions via an unsupervised procedure. We introduce these methods in the next part.

#### 5.2.1 Supervised TN Models

Supervised models are used to model the conditional probability distributions of labels (output) given input features based on example input-output pairs. Taking embedded quantum data as inputs, Stoudenmire and Schwab [105] proposed supervised MPS-like tensorial multilinear models and adopted DMRG-like algorithms to optimize the model weights. Prior to introducing a specific implementation, their models must first be formulated as procedures that optimize a set of functions indexed by different labels \(\ell\):

\[f^{\boldsymbol{\ell}}(\mathbf{x})=\boldsymbol{\mathcal{W}}^{ \boldsymbol{\ell}}\times_{1,2\cdots N}^{1,2\cdots N}\Phi(\mathbf{x}), \tag{38}\]

where \(\Phi(\cdot)\) is the feature map function in Eq. (36) and \(\boldsymbol{\mathcal{W}}^{\boldsymbol{\ell}}\in\mathbb{R}^{\overbrace{2\times 2 \cdots\times 2}^{N}}\) is the weight tensor. Then, the aforementioned tensorial models can be derived by replacing \(\boldsymbol{\mathcal{W}}^{\boldsymbol{\ell}}\) with an MPS TN:

\[\boldsymbol{\mathcal{W}}^{\boldsymbol{\ell}}_{s_{1}s_{2}\cdots s_ {N}}=\sum_{\{\alpha\}}\mathcal{A}^{(1)}_{s_{1},\alpha_{1}}\cdots\mathcal{A}^{( m)}_{\alpha_{m-1},s_{m},\alpha_{m}}\cdots\mathcal{A}^{(N)}_{\alpha_{N-1},s_{N}}, \tag{39}\]

where \(\mathcal{A}^{(1)}\cdots\mathcal{A}^{(N)}\) are core tensors. Furthermore, the authors proposed an optimization algorithm named Sweeping that was motivated by the DMRG algorithm [8] in quantum mechanics. The optimization algorithm sweeps along an MPS to optimize the quadratic cost function:

\[c=\frac{1}{2}\sum_{n=1}^{N_{T}}\sum_{\ell}\left(f^{\ell}\left( \mathbf{x}_{n}\right)-\mathbf{y}_{n\ell}\right)^{2}, \tag{40}\]

Fig. 10: The processing procedure employed for quantum embedded data [183]. Quantum circuits can be simulated via TNNs on classic electronic computers, and some special TNNs (such as ConvAC) can also be theoretically implemented on a realistic quantum circuit.

Fig. 9: TN diagrams of PTP. CP and TR structures can be adopted in such a strategy.
where \(N_{T}\) denotes the number of training samples, and \(\mathbf{y}_{n}\) denotes the true one-hot label vector of \(\mathbf{x}_{n}\). The optimization process is carried out to minimize this cost function in stages with stochastic gradient descent. A single stage is shown in Fig. 11. In each stage, two MPS tensors \(\mathbf{\mathcal{A}}^{(2)}\) and \(\mathbf{\mathcal{A}}^{(3)}\) are combined into a single bond tensor \(\mathbf{\mathcal{V}}\) via tensor contraction. Then, the tensor \(\mathbf{\mathcal{V}}\) is updated with gradients. Finally, \(\mathbf{\mathcal{V}}\) is decomposed back into separate tensors with the SVD algorithm. The Sweeping method is efficient in optimizing models whose inputs embedded quantum data, and it can also be adopted to train TNNs. In addition, other TNs, including the PEPS structure [186], can be processed with a Sweeping-like method.

#### 5.2.2 Unsupervised TN Models

The goal of unsupervised generative modeling is to model the joint probability distribution of the given data. Generative adversarial networks (GANs) [187] and variational autoencoders (VAE) [188] are successful NN models for addressing classic data distributions. Embedded quantum data are the key to training and designing quantum TNNs for generating probabilistic distributions via TNs. Inspired by the probabilistic interpretation of quantum states in quantum mechanics [189], an MPS-based generative model called the Born machine [185] was proposed. The Born machine is an energy-based model [190] derived from quantum mechanics. The distribution functions of the Born machine are shown as follows:

\[P(\mathbf{x})=\frac{|\Psi(\mathbf{x})|^{2}}{Z}, \tag{41}\]

where \(Z=\sum_{x}|\Psi(x)|^{2}\) is the normalization factor, \(\Psi(\cdot)\) is the quantum state embedding function, and the energy function of \(\mathbf{x}\) can be represented as \(|\Psi(\mathbf{x})|^{2}\) in view of quantum mechanics. \(\Psi(\mathbf{x})\) can be parameterized via a TN format [185]. The learning procedure can also be conducted via a DMRG-like algorithm and a gauge transformation strategy [191]. The distribution definitions in Eq. (41) are also useful for designing the loss functions of quantum TNNs. Furthermore, a series of Born machine structures have been proposed, including tree TNs [109], uniform MPSs for language generation [108], and locally purified states [110]. In the future, it is expected that representing and processing data via a particular quantum view will pave the way for the further implementation of TNNs on realistic quantum computers [109].

### _ConvAC Network_

The expressive power of previously developed quantum data processing models, e.g., the MPS models in Section 5.2.1 and the Born machine in Section 5.2.2, suffers from a lack of nonlinearity. Classic nonlinear operators, e.g., activation functions (such as the rectified linear unit (ReLU) function) and average/max pooling, can significantly benefit model performance. However, classic nonlinearity cannot be directly implemented in a quantum circuit. To solve this problem, the ConvAC network [111, 192] was proposed to adopt quantum deployable product pooling as a nonlinear operator, proving that ConvAC can be transformed into ConvNets with ReLU activations and average/max pooling.

The whole structure of ConvAC can be represented by an HT format and has been proven to be theoretically deployable in realistic quantum systems. A tensor diagram example of ConvAC is shown in Fig. 12, and one hidden layer of ConvAC is in a CP format. ConvAC can also handle language data [107] by mapping natural language sentences into quantum states via Eq. (37). ConvAC is a milestone in that deep convolutional networks, along with nonlinear modules, are implemented on quantum circuits. It serves as an inspiration for the integration of more NNs into quantum systems. For instance, Zhang et al. [112] introduced the tensor space language model (TSLM), which has been shown to be a generalization of the n-gram language model.

**Remark.** The implementation of quantum TNNs is a symbolic milestone that forms a bridge between QNNs and classic NNs via TNs. However, strict mapping algorithms between simulated quantum TNNs and realistic physical systems still need to be explored. Moreover, as real high-performance quantum computers are still a long way from being developed, the concept of verifying the performance of quantum TNNs in the near future is infeasible. Despite these issues, it is still important and meaningful to focus on mapping algorithms and performance verifications for QNNs.

## 6 Training Strategies for TNNs

While the aforementioned TNNs can perform well on various tasks and machines, it is also worth exploring training strategies with more stability, better performance and higher efficiency. In this section, we introduce such strategies in three groups; (1) strategies for stabilizing the training processes of TNNs are presented in Section 6.1, (2) strategies for selecting and searching the ranks of TNNs are provided in Section 6.2, and (3) strategies for applying hardware speedup are shown in Section 6.3.

### _Stable Training Approaches_

Despite a variety of successes, TNNs still suffer from training problems due to their multilinear characteristics. Compared to

Fig. 11: A single stage of the Sweeping method [105]. In each stage, Sweeping only updates the nodes in the sweeping window, which shifts along a zigzag trajectory.

Fig. 12: ConvAC is equivalent to an HT-like TN [180]. \(\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\in\mathbb{R}^{s}\). \(\mathbf{x}_{1}\) corresponds to a local patch from the input image or the feature map, and \(v^{(0,j)}\) is the linear transformation of \(\mathbf{x}_{j}\). The width is 2 for a single block. Notably, a single layer is equivalent to a CP format.
simple linear operations such as matrix production, tensor contraction yields data flows with exponential scales, i.e., features in forward propagation and gradients in backward propagation, when modes linearly increase [114]. One solution is to utilize the full-precision float64 format to denote large weights, which can alleviate these numerical issues to some extent. However, a full-precision format can result in more calculations and higher time consumption compared to a lower-precision format, e.g., float16. Nevertheless, low precision may cause numerical stability issues. To solve these problems, Panagakis et al. [113] proposed a mixed-precision strategy to form a tradeoff. This dynamic precision strategy is efficient in reducing memory occupation and promotes training stability.

Another feasible way to solve the training problem lies in developing a suitable initialization.Commonly used adaptive initialization methods include Xavier [195] initialization and Kaiming [196] initialization, which regulate the variances of the data flows in layers. However, these two initializations cannot calculate the correct scales of TNNs to neglect interactions in tensor contractions. Furthermore, tensor formats are different from each other, causing difficulty in developing a general fitting initialization for diverse tensorial layers. To solve these two problems, Yu initialization [114] proposed a unified initialization paradigm based on Xavier to adaptively initialize for arbitrary TCNNs. Specifically, Pan et al. extracted a backbone graph (BG) from a tensorial convolution hypergraph [61] and then encoded an arbitrary TCNN into an adjacency matrix with this BG. Finally, a suitable initial variance for a TCNN can be directly calculated through the adjacency matrix. We illustrate three cases of using unified initializations in Fig. 13. Although Yu initialization was proposed for TCNNs, it can also be widely used in other NNs, including CNNs, RNNs, and Transformers, since these models basic layers, i.e., convolutional and linear layers, belong to the scope of TCNNs.

### _Rank Selection and Search_

Prior studies [67, 69, 79] focused on finding efficient TN formats (e.g., TTs and TRs) for compressing NNs and achieve significant efficiency for their natural compact structures. However, despite these remarkable successes, efficient algorithms for adjusting or selecting suitable ranks for a TN are lacking since rank selection is an NP-hard problem [153]. As a result, many approaches [64, 68, 77, 69] can only set values for all ranks manually, which severely affects the resulting models' training procedures. Fortunately, the rank selection problem can still be optimized through heuristic strategies, such as Bayesian optimization [118], reinforcement learning (RL) [116] and evolutionary algorithms (EAs) [115]. Here, we introduce some rank selection methods for TNNs.

DNNs utilize neural architecture search (NAS) [197] to search for the optimal network hyperparameters, achieving significant success. As ranks can be treated as architecture hyperparameters, NAS is applicable to searching for optimal tensorial layers with better rank settings. Following this idea, the progressive searching TR network (PSTRN) [115] employs NAS with an EA to select suitable ranks for a TR network (TRN). In detail, the PSTRN employs a heuristic hypothesis for searching: "when a shape-fixed TRN performs well, part or all of its rank elements are sensitive, and each of them tends to aggregate in a narrow region, which is called an interest region". Instructed by the interest region hypothesis, the PSTRN can reach the optimal point with a higher probability than a plain EA method. The PSTRN consists of an evolutionary phase and a progressive phase. During the evolutionary phase, this method validates the ranks in the search space on benchmarks and picks the rank that yields the best performance. Then, in the progressive phase, the PSTRN samples new ranks around the previously picked rank and inserts them into a new search space. After several rounds, the heuristic EA can find a high-performance solution. With such an efficient design, the PSTRN successfully achieves better performance than hand-setting, which demonstrates that its hypothesis is practical.

In addition to NAS, some other efficient methods are also available for rank selection. Zhao et al. [117] inferred a CP rank by implementing a reduction process on a large rank value via a variational Bayesian optimization procedure. Hawkins and Zhang [118] extended this CP procedure [117] to TT-based TNNs and adopted the Stein variational gradient descent method, which combines the flexibility of the Markov chain Monte Carlo (MCMC) approach with the speed of variational Bayesian inference to construct a Bayesian optimization method. In pretrained networks, Kim et al. [120] and Gusak et al. [121] derive approximate ranks by employing Bayesian matrix factorization (BMF) [198] to unfolding weight tensors. Unlike Bayesian methods, Cheng et al. [116] treated the rank searching task as a game process whose search space was irregular, thus applying RL to find comparably suitable ranks for a trained CNN. However, this algorithm is TD-dependent, which indicates that its performance may be influenced by the selected TD method. Yin et al. [119] leveraged the alternating direction

Fig. 13: Three cases of unified TCNN initialization [114]. \(\sigma^{2}\) denotes the initial variance of each weight vertex. \(\mathbf{G}_{f}\) denotes a forward procedure, and \(\mathbf{G}_{b}\) denotes a backward procedure. (i) Standard convolution. The method in [114] degenerates to Xavier/Kaiming initialization on the standard convolution for the same weight variance formulation. (ii) Hyper Tucker-2 (HTK2) convolution. Tucker-2 (TK2) is a common TD that is utilized in ResNet as the bottleneck module [193]. HTK2 is formed by applying a hyperedge to the weight vertices of TK2. (iii) Odd convolution. The odd TD was originally proposed by [194]. The connections among the vertices are irregular, making weight initialization a complex problem. These three successful initialization cases can better demonstrate the potential adaptability of unified initialization to diverse TCNNs.
method of multipliers (ADMM) to gradually transfer the original weight to a low-rank representation (i.e., a TT).

### _Hardware Speedup_

Accelerating the training and inference procedures of TNNs can benefit resource consumption and experimental adjustment, thereby achieving economic gains and green research. A direct and effective approach is to optimize the speed of tensor operations in TNNs to realize hardware acceleration. As inferring TT-format TNNs inevitably results in enormous quantities of redundant calculations, the TIE scheme [122] was proposed to accelerate TT layers by splitting the working SRAM into numerous groups with a well-designed data selection mechanism. Huang et al. [123] designed a parallel computation scheme with higher I/O bandwidth, improving the speed of tensor contractions. Later, they proposed an LTNN [123] to map TT-format TNNs into a 3D accelerator based on CMOS-RRAM, leading to significantly increased bandwidth via vertical I/O connections. As a result, they simultaneously attained high throughput and low power consumption for TNNs. Recently, Qu et al. [124] proposed a spatial 2D processing element (PE) array architecture and built a hardware TT engine consisting of off-chip DRAM. Kao et al. [125] proposed an energy-efficient hardware accelerator for CP convolution with a mixing method that combines the Walsh-Hadamard transform and the discrete cosine transform.

Many more fascinating methods have been developed for the acceleration of generic tensor operations, which are correlated with TNNs. For instance, Huang et al. [199] observed that the tensor matricization operation is usually resource-consuming since its DRAM access is built on a random reading address; thus, they proposed a tensor storage scheme with a sequential address design for better DRAM accessibility. Both T2s-tensor [200] and Tensaurus [201] mainly focus on designing general computation kernels for dense and sparse tensor data. Xie et al. [149] and Liang et al. [202] accelerated search procedures for obtaining an optimal sequence of tensor contractions. Xie et al. [149] solved the massive computational complexity problem of double-layer TN contraction in quantum analysis and mapped such a double-layer TN onto an intersected single-layer TN. Liang et al. [202] implemented multithread optimization to improve the parallelism of contractions. Fawzi et al. [203] also illustrated the potential of RL to build efficient universal tensor operations. In the future, it is expected that more general hardware acceleration schemes based on tensor operations will be developed to implement TNNs with smaller storage and time consumption levels.

**Remark.** The comments are divided into three parts. (1) To achieve training stability, it is possible to borrow ideas concerning identity transition maintenance to construct more stable initializations. In addition, it is also feasible to add adversarial examples to enhance network robustness. (2) Rank search is important for further improving the performance of TNNs. However, as it is an NP-hard problem, rank search has not been sufficiently explored. In the future, suitable ranks can be searched through the guidance of gradient sizes and EAs in searching for TNN architectures. (3) Last, research on hardware has derived some success in terms of speed acceleration and memory reduction. However, these methods are almost ad hoc designs for specific TD formats, so they lack applicability to other TNN structures.

## 7 TNN Toolkoves

In 1973, Pereyra and Scherer [204], as pioneers in this field, developed a programming technique for basic tensor operations. Recently, with the development of modern computers, many more basic tensor operation toolboxes have been developed, and a series of powerful TNN toolboxes have also been proposed for both network compression and quantum circuit simulation, which are the two main applications of TNNs. In this section, toolboxes for TNNs are presented in three categories according to their design purposes: (1) toolboxes for basic tensor operations contain important and fundamental operations (e.g., tensor contraction and permutation) in TNNs (Section 7.1); (2) toolboxes for network compression are high-level TNN architecture toolboxes based on other basic operation tools (Section 7.2); and (3) toolboxes for quantum circuit simulation are software packages for the quantum circuit simulation or quantum machine learning processes that use TNs from a quantum perspective (Section 7.3).

### _Toolboxes for Basic Tensor Operations_

Toolboxes for basic tensor operations aim to implement some specific TD algorithms. Many basic tensor toolboxes based on different programming languages and backends have been designed for this purpose. For example, The online stochastic framework for TD (OSTD) [130] and Tensor Toolbox [128] were constructed for low-rank decomposition and implemented with MATLAB. Regarding Python-based toolboxes, TensorTools based on NumPy [205] implements CP only, while T3F [136] was explicitly designed for TT decomposition on TensorFlow [206]. Similarly, based on TensorFlow, TensorD [131] supports CP and Tucker decomposition. Tntorch [133] is a PyTorch-based library for tensor modeling in the CP, Tucker and TT formats. TorchMPS [134], TT-Toolbox [132] and Scikit-TT [138] are all powerful Python-based specific TT solvers that efficiently implement the DMRG algorithm. Tensorly is a powerful general TD library that supports many decomposition formats and various Python backends including CuPy, Pytorch, TensorFlow and MXNet [207]. TensorNetwork [137] is a powerful general-purpose TN library that supports a variety of Python backends, including JAX, TensorFlow, PyTorch and NumPy. In addition, some toolboxes based on C++ are also available. TenDec++ [208] leverages a unique pointer technology called PointerDeformer in C++ to support the efficient computation of TD functions. ITensor [135] is an efficient and flexible C++ library for general TN calculations.

### _Toolboxes for Network Compression_

Specific TNN toolboxes are used to assist with the development of tensorial layers. Although some general tensor toolboxes such as Tensorly [126] are powerful for TD processing and can use their TD operations to help initialize TNN modules to a certain extent, they still lack support for application programming interfaces (APIs) for building TNNs directly. Therefore, a TNN library (Tensorly-Torch) based on Tensorly was developed to build some tensor layers within any PyTorch network. Pan et al. also developed a powerful TNN library called TedNet [64]. TedNet can quickly set up TNN layers by directly calling the API. In addition, TedNet supports the construction of TCNNs and TRNNs in single lines of code.

### _Toolboxes for Quantum Circuit Simulation_

A number of quantum circuit simulation toolboxes have been designed. For example, some TT toolboxes such as Scikit-TT and TorchMPS can partially simulate quantum circuits to some extent, although they were not specifically designed for quantum circuit
simulation. In contrast, general TN toolboxes, e.g., TensorNetwork and ITensor, can simulate any quantum circuit. In addition, with optimized tensor contraction, TeD-Q [141], a TN-enhanced open-source software framework for quantum machine learning, enables the simulation of large quantum circuits. Furthermore, Yao [139], an extensible and efficient library for designing quantum algorithms, can provide support for dumping a quantum circuit into a TN. Although no practical implementations of quantum TNNs are available, these quantum circuit simulations are potentially useful for the simulation of quantum TNNs.

**Remark.** Despite the success of current toolboxes, some areas for improvement remain. (1) Existing basic tensor operation toolboxes are built using high-level software frameworks, limiting their ability to fully utilize the inherent capability of tensor computations. (2) Existing deep model implementation toolboxes for TNNs can only contain a limited number of predefined TNN structures and cannot allow users to design structures freely. (3) Existing quantum simulation toolboxes focus more on the simulation of quantum circuits using TNs and do not facilitate the processing of embedded quantum data via TNNs.

## 8 Conclusion and Future Prospects

This survey embraces the connection between TNs and NNs, summarizing the techniques for compressing NN parameters, modeling the interactions between different dimensions of data, and bridging QNNs with classic NNs. As previously noted, TNNs have considerable strengths and immense potential, making them applicable in many areas. In the future, research on TNNs can benefit from the optimization and implementation of tensor-friendly hardware; we believe that TNNs will have large impacts on studies involving quantum physics and quantum computers.

**Acceleration based on hardware design.** Although many TNNs have low calculation complexity levels in theory, realistic hardware deployments usually fall short of this objective due to their numerous permutation operations [64] and the absence of sufficient parallelism [123]. Efficient hardware and matching software for universal tensor acceleration or TNN acceleration can be developed. As mentioned in Section 6.3, the existing tensor acceleration software and hardware structures are all aimed at certain TN structures or are based on parallel matrix acceleration. The acceleration of general TN operations is necessary and urgent for the implementation of TNNs.

**Applications in quantum physics.** In some specific physical applications that need to deal with large-scale tensors, such as wave function simulation [209], specifically designed TNNs can be studied to efficiently solve these problems involving higher-order interactions. Inspired by the universal approximation theorem, some simple NNs have been adopted in wave function simulation tasks, such as free boson and fermion systems [210]. However, because of the curse of dimensionality, simple NNs are difficult to apply in extremely large-scale wave function simulation tasks, and TNNs can easily handle tasks with large-scale tensor as a result of the compact nature of TNs.

**Implementations in quantum mechanics.** The existing TNNs mainly adopt the mathematical forms of TNs and seldom consider the physical properties of the quantum systems described by these TNs [113, 112]. Since TNNs are highly related to quantum circuit structures, it is possible to obtain more efficient and effective TNNs by applying the concepts and theory of TNs in quantum mechanics. For example, optimizing and interpreting TNNs from the perspective of entanglement entropy theory [211] could be a meaningful direction for producing interpretable and efficient NNs.

## Acknowledgments

This paper was partially supported by National Key Research and Development Program of China (No. 2018AAA0100204), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), and the Major Key Project of PCL (No. PCL2021A06).
