Unified Framework for Neural Network Compression via Decomposition and Optimal Rank Selection
Ali Aghababaei-Harandi Massih-Reza Amini


Communication-Assisted Sensing Systems: Fundamental Limits and ISAC Waveform Design
    Fuwang Dong, Member, IEEE, Fan Liu, Senior Member, IEEE, Yifeng Xiong, Member, IEEE, 
 Yuanhao Cui, Member, IEEE, Wei Wang, Senior Member, IEEE, Shi Jin, Fellow, IEEE 

(Corresponding author: Fan Liu.)
Part of this paper was presented at IEEE International Symposium on Information Theory (ISIT), 2024 <cit.>.

Fuwang Dong, and Fan Liu are with the School of System Design and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen 518055, China. (email: {dongfw, liuf6}@sustech.edu.cn).

Yifeng Xiong, and Yuanhao Cui are with the School of Information and Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China. (email: {yifengxiong, cuiyuanhao}@bupt.edu.cn).

Wei Wang is with the College of Intelligent System Science and Engineering, Harbin Engineering University, Harbin 150001, China. (email: wangwei407@hrbeu.edu.cn).

Shi Jin is with the National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China. (e-mail: jinshi@seu.edu.cn). 
    September 9, 2024
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Despite their high accuracy, complex neural networks demand significant computational resources, posing challenges for deployment on resource-constrained devices such as mobile phones and embedded systems. Compression algorithms have been developed to address these challenges by reducing model size and computational demands while maintaining accuracy. Among these approaches, factorization methods based on tensor decomposition are theoretically sound and effective. However, they face difficulties in selecting the appropriate rank for decomposition. This paper tackles this issue by presenting a unified framework that simultaneously applies decomposition and optimal rank selection, employing a composite compression loss within defined rank constraints. Our approach includes an automatic rank search in a continuous space, efficiently identifying optimal rank configurations without the use of training data, making it computationally efficient. Combined with a subsequent fine-tuning step, our approach maintains the performance of highly compressed models on par with their original counterparts. Using various benchmark datasets, we demonstrate the efficacy of our method through a comprehensive analysis.





§ INTRODUCTION


In recent years, deep learning has revolutionized various scientific fields, including computer vision and natural language processing <cit.>. Complex neural networks, with often millions or even billions of parameters, have achieved unprecedented levels of accuracy across a diverse array of tasks. However, this exceptional performance comes at a cost in terms of computational resources. The immense size of these state-of-the-art neural networks presents a challenge for their deployment on devices and platforms with limited resources, such as mobile phones, edge devices, and embedded systems <cit.>. The storage, memory, and processing requirements of these models often prove to be unfeasible or excessively costly, thus limiting their practicality and accessibility.

Recent research  have introduced a range of compression algorithms aimed at addressing challenges related to cost-effectiveness, scalability, and real-time responsiveness <cit.>. These studies can be classified into four primary categories of approaches, all of which effectively reduce a model's size and computational demands while preserving its accuracy. One of the most straightforward methods is pruning, which involves identifying and removing insignificant weights from the model <cit.>. Quantization, on the other hand, focuses on reducing the precision of numerical values in the model, typically transitioning from 32-bit floating-point numbers to lower bit-width fixed-point numbers <cit.>. Knowledge distillation  is another technique where a smaller model, often termed the “student”, is trained to mimic the behavior of a larger model, known as the “teacher” <cit.>. This knowledge transfer results in the creation of a smaller model capable of approximating the performance of the larger one. Lastly, factorization methods partition the weight matrices or tensors of a neural network into smaller matrices or tensors, effectively reducing the number of parameters in the model <cit.>. While factorization techniques  have proven to be effective and efficient in reducing model size, a significant challenge lies in the selection of the appropriate rank for the decomposition process. 

Non-uniqueness in tensor rank is a critical challenge in tensor decomposition research. Most tensor decomposition problems, particularly CP decomposition, are NP-hard, as noted by <cit.>. This non-uniqueness arises from ambiguities in the factorization and scaling process, allowing different decompositions to produce the same original tensor. Finding the ideal rank remains an ongoing research topic, and determining multiple tensor ranks for the weights of different layers in  deep neural networks is not well suited for conventional hyperparameter selection techniques like cross-validation. As a result, it is standard practice to choose a single rank for all decompositions within a task based on a compression rate. However, this simplification can lead to significant performance degradation specially in complex models.


Recent studies propose automated methods for determining the ranks of decomposition <cit.>. However, many of these approaches, including reinforcement methods, greedy search algorithms, and SuperNet search, can be computationally expensive and time-consuming, particularly for large models and datasets. Moreover, the effectiveness of automated rank selection methods often depends on the choice of hyperparameters, such as learning rates or regularization parameters, which can be challenging to tune. Additionally, none of these existing approaches cover a wide range of search space, preventing the achievement of ideal compression rates.




In this paper, we address these challenges by introducing a unified framework named Optimal Rank Tensor decOmpoSition (), which simultaneously tackles both the decomposition and optimal rank selection tasks. This is achieved through the utilization of a composite compression loss within specified rank constraints. Also, when we combine this rank search with a subsequent fine-tuning step, our experiments demonstrate that the performance of the resulting highly compressed model remains on par with the original model. Overall, the contributions of this paper is summarized as follows:



  *  allows  to achieve maximum compression rates by covering all ranks in the search space through a simple and efficient multi-step search process that explores ranks from low to high resolution.


  * The proposed search method involves an automatic rank search in a continuous space, which efficiently identifies the optimal rank configurations for layer decomposition without requiring data, making it computationally efficient.


  * We perform a comprehensive analysis of the various components of our approach, highlighting its efficacy across various benchmark datasets. we achieved improvement in some experiments specifically improvement in all metrics in the case of ResNet-18, while in another experiment we had competitive results. Moreover, our method speeds up the search phase compared to other related work.



To our knowledge, this is the first effort to use bilevel optimization to find the optimal ranks of a pre-trained model. 




§ RELATED WORKS

Decomposition techniques, despite their apparent simplicity, have garnered considerable attention in the field of deep learning, particularly in natural language processing (NLP) <cit.>. They provide an efficient means to fine-tune large language models, offering notable advantages over alternative methods such as integer-based compression <cit.>, knowledge distillation <cit.>, and gradient-based pruning <cit.>. Despite their straightforward nature, decomposition approaches serve as a robust compression tool with a high compression rate and relatively lower computational cost. Their application extend beyond NLP, finding utility in computer vision models <cit.>. However, selecting the appropriate rank for compressing deep neural models using decomposition techniques is an NP-hard challenge <cit.>. Research in this domain can be categorized into two main approaches. 

First approach is a rank-fixed setting, where the ranks of layers are determined based on a predefined compression rate target. Employ approach for the pre-trained mode was initially explored by <cit.>, which employed a low-rank loss to substitute the weights of convolution layers with their low-rank approximations. Specifically, two main low-rank approximation methods, namely CP and Tucker decomposition, are utilized to break down layers of the pre-trained models <cit.>. Moreover, <cit.> discovered unstability of fine-tuning after CP decomposition and address this issue by introducing a stability term into the decomposition process. 

In addition, some researchers design decomposed models to train from scratch. <cit.> demonstrated the compression of deep models through redesign convolution and fully connected layers with a tensor train. Furthermore, LSTM models, which face computational challenges in vision tasks, seek to mitigate by employing tensor train and tensor ring decomposition<cit.>. <cit.> use the potential of tensor rings to compress the model by replacing layers with its equal decomposition of the tensor ring. From another perspective,<cit.> proposed a novel rank constraint training scheme to the train model to achieve inherently low-rank layers. With attention to the fact that transformers have huge parameters, <cit.> 
tensorized pre-trained language models with matrix factorization and tensor decomposition.




However, a technical analysis of the rank-fixed setting approach uncovers certain challenges. Firstly, selecting the appropriate rank for layers relies on human expertise, introducing a potential bottleneck. Secondly, there is a lack of interpretable patterns between layer ranks, resulting in a disconnect among chosen ranks across layers. These limitations occasionally lead to accuracy drops or insufficient compression rates.

The second approach involves determining the optimal ranks by framing the optimization problem based on the ranks of layers. <cit.> introduce an iterative method to gradually decrease the ranks of the layers in each step of the search. In particular, the discrete nature of the rank search prompts researchers such as <cit.> to employ discrete search algorithms such as reinforcement learning and progressive search to identify optimal ranks. In addition, <cit.> compress the model by imposing constraints on ranks and budget, employing an iterative optimization strategy. Furthermore, in exploring the ranks of transformer layers, <cit.> propose a two-step search process through evolutionary search techniques to determine the optimal rank.



Moreover, recent studies <cit.> aim to utilize a continuous search space to determine optimal ranks, as our proposed approach. There are three main differences between our method, , and the theirs. First, these approaches search for ranks through the training of a large decomposed SuperNet from scratch, incurring substantial computational expenses. whereas, we utilize a pre-trained network to ascertain optimal layer ranks. Secondly, we propose a straightforward and effective loss function that operates independently of the data, thus speeding up our search process for large models. Finally, we leverage engineering insights to partition our search space into smaller search steps, encompassing a wider range of ranks, while, they limit their exploration to a constrained search space of ranks, hindering to achieve high compression rate.




§ BACKGROUND AND PRELIMINARIES

In this section, we present notation and background relevant to our approach.


  
Notation. We represent indices using italicized letters, sets with italic calligraphic letters and tensors as multidimensional arrays with bold calligraphic script letters. For two-dimensional arrays (matrices) and one-dimensional arrays (vectors), we use bold capital letters and bold lowercase letters, respectively.


  
Decomposition. Tensor decomposition is a method that transforms a multi-dimensional array of data into a series of lower-dimensional tensors, thereby reducing both the data size and computational complexity. Singular value decomposition (SVD) stands as the most basic form of decomposition, typically applied to two-dimensional matrices. the generalized concept of SVD is used for general tensors, known as tensor decomposition. The prevalent tensor decomposition techniques encompass canonical polyadic (CP), Tucker, tensor train (TT) and tensor ring (TR) decomposition <cit.>.

Without loss of generality, this paper explores CP and TT  for tensor decomposition.  CP decomposition entails breaking down a given weight tensor W∈ℝ^I_1× I_2×⋯× I_N into a series of rank-one tensors which can mathematically be represented  as follows:


    Ŵ^(R)(i_1,i_2,...,i_N)     = ∑_r=1^R_̌1^(r)(i_1) _̌2^(r)(i_2) ⋯_̌N^(r)(i_N) 
       =_1,_2,...,_N


Where Ŵ^(R) represents the decomposition of W of rank R, and ^̌(r)_n are vectors of the same size as the dimension I_n of W, and,  _̌n^(r)(i_n) represents the i_n^(th) element of the r^th factor vector for the n^th mode.  _1,_2,...,_N is the Kruskal operator which represents the tensor constructed from the matrices _1,_2,...,_N where each matrix _i contains the vectors ^̌(r)_i as its column.

TT decomposition also decomposes a tensor into smaller tensors with dimensions connected like a chain to each other. This decomposition mathematically can be represented as follows:


    Ŵ^(R_1,…,R_N-1)(i_1,i_2,...,i_N) =  ∑_j_1=1^R_1⋯∑_j_N-1=1^R_N-1G_1(i_1,j_1)
    G_2(j_1,i_2,j_2) ⋯G_N(j_N-1,i_N)


where the tuple (R_1,R_2,…,R_N-1) represents the rank of the TT decomposition, and 𝒢_k are the TT cores with sizes R_k-1× I_k× R_k, and R_0=R_N=1. 
A visualization of these methods for a three-dimensional tensor is provided in Figure <ref>.

Layer decomposition representation. In our study, we focus on the decomposition of convolutional layers, which, when stacked, form a tensor. Given a convolutional layer with a weight tensor  W∈R^b× h× w× c, the forward process for an input tensor X∈R^k_1× k_2× k_3 can be expressed as:


    Y =∑_i_1=0^k_1-1∑_i_2=0^k_2-1∑_i_3=0^k_3-1W(t,x+i_1,y+i_2,z+i_3) X (i_1,i_2,i_3)


 Specifically, we investigate how the weight tensor of a convolutional layer can be decomposed into multiple smaller convolution operations. We utilize CP and TT decompositions, as detailed in the following formulations.


    
  * CP decomposition:


    Y   =∑_r=1^R_t(t,r)( ∑_i_1=0^k_1-1_x(x+i_1,r) (∑_i_2=0^k_2-1_y(y+i_2,r) 
       (∑_i_3=0^k_3-1_s(z+i_3,r) X (i_1,i_2,i_3)) ) )




    
  * TT decomposition



    Y   =∑_r_1=1^R_1∑_r_2=1^R_2G_t(t,r_1) (∑_i_1=0^k_1-1∑_i_2=0^k_2-1G_y(r_1,x+i_1,y+i_2,r_2) 
        (∑_i_3=0^k_3-1G_s(i_3,r_2) X (i_1,i_2,i_3)) )



§ OPTIMAL RANK TENSOR DECOMPOSITION


Given a pre-trained neural network with n hidden layers and convolutional weights (W_i)_i=1^n, our objective is to achieve a low-rank decomposition of these weights with the smallest possible ranks and which can be formulated as the following optimization problem:


    W^min ℒ_d(W^)     s.t.   R_1,…, R_nmin∑_i=1^n R_i


where ℒ_d(.) is a decomposition loss function, and W^= {Ŵ^(R_1)_1,…, Ŵ^(R_n)_n } is the set of decompositions to be found with ={R_1,…,R_n} the set of ranks, and Ŵ^(R_i)_i the decomposition of rank R_i corresponding to the convolutional weight W_i of layer i. Note that in state-of-the-art compression models using decomposition techniques, the ranks of the decompositions for each convolutional weight are typically treated as hyperparameters and are fixed. In contrast, our study proposes to automatically determine these ranks along with the decompositions.




 §.§ Problem Formulation

In Eq. <ref>, the constraint on the ranks is to select the lowest possible rank for each tensor decomposition while maintaining minimal decomposition error.  To achieve this, we replace each layer i∈{1,…,n} of the network with a super layer, which contains  decomposed weights of various ranks from a given set _i, denoted as (Ŵ_i^(r))_r∈_i, corresponding to the convolutional weight of that layer, W_i. The structure of this super layer is illustrated in Figure <ref> (left). To make continuous equation (<ref>), we associate a probability p_i^(r) with each decomposition of rank r in layer i based on a learnable parameter α_i^(r). This probability is computed as p_i^(r)=softmax(α_i^(r)), which plays a role in the optimization process by serving as a coefficient for the rank. Inspired by <cit.>, the rank constraint in equation (<ref>) is formalized by the following loss:


    _R =∑_i=1^n(∑_r∈_i p_i^(r) r    )^β

Where β∈[0,1] is a hyperparameter. Traditional methods often rely on using data for model compression via tensor decomposition techniques <cit.>. In contrast, our approach optimizes without directly using data. Instead, we minimize the mean-square decomposition error (MSDE), measured by the Frobenius norm, between the convolutional tensor weights of the pre-trained model and their corresponding tensor approximations, as defined by equation (<ref>) for CP decomposition or (<ref>) for TT decomposition. This method aims to accelerate the convergence of the optimization process by simplifying the objective to a weight-based measure rather than data-driven metrics. After determining the decompositions, we fine-tune the model using data to mitigate approximation errors and preserve performance. Our total loss for a model with n layers is as follows:


    ℒ_T = ∑_i=1^n[𝒲_i-∑_r∈_i  p_i^(r)𝒲̂^(r)_i^2_F  + γ(∑_r∈_i p_i^(r) r)^β]

For all layers, the decomposition loss in the left term of equation (<ref>) is designed to minimize the approximation error within their corresponding super-layers, while the rank loss in the right term aims to maximize the probability of selecting the optimal super-layer for each layer. This combination introduces a trade-off, where the objective of finding the best approximations competes with the goal of selecting the lowest possible ranks. The hyperparameter γ expresses this trade-off. In the next section, we will present an approach that examines all possible ranks within a broad search space for an optimal compression rate.




 §.§ Rank Search Space



We define the rank search space as a collection of rank sets {_1,…,_n}, where each set _i for i∈{1,…,n} contains the rank values to be explored for decomposing the weight tensor of layer i.

To efficiently explore the entire search space and find the optimal rank, we propose a multi-step search strategy that divides the space of possible ranks into smaller regions and progressively refines the search resolution. The process begins with an initial pruning phase, where specific ranks are selected at regular intervals for each layer, reducing the initial number of ranks under consideration. At this stage, promising candidate ranks are identified, narrowing down those most likely to be optimal.

Following this stage, a new search space is generated, focused around the selected ranks (SR) from the previous step, but with smaller intervals between them. This allows for a more precise examination of ranks in a concentrated region. The process is repeated, with the intervals between ranks gradually decreasing in each iteration. This progressive refinement continues until the interval between ranks is reduced to one, ensuring that all possible ranks within the search space have been explored. 




Figure <ref> depicts this process for one layer. The initial search space spans a broad interval with a larger step size. After the first iteration, the process narrows down to a more focused interval around the selected rank, reducing the step size by dividing it by 10. Following the second iteration, the search space is further refined to a smaller, more precise interval with a step size of one for the final selection. To facilitate this process, a function named the rank search space  function (RSS-function) is proposed. This function generates a search rank set for each layer. It takes the lower and upper bounds for the ranks, as well as a step value and the number of layers as inputs, and produces a list of ranks that define the search rank set for each layer at each iteration of the algorithm. The pseudocode of this function is provided in the supplementary material.




 §.§ Tensor Decomposition and Rank Exploration

The minimization of the total loss (<ref>) is achieved by updating tensor weights Ŵ_i^(r) of each layer i∈{1,…,n} and the probability parameters α_i^(r) for different ranks in the search rank set r∈_i corresponding to layer i. 

Most of the computational cost of optimizing this loss function is concentrated on the decomposition component, which minimizes the squared error between the decomposed layer weights and the original layer weights. By using stochastic gradient descent (SGD) instead of the classical alternating least squares (ALS) method, we can expect fast convergence, enabling to repeat this process multiple times. Each iteration of our optimization includes the following two steps:

    Weight update: Ŵ^(r)_i   Ŵ^(r)_i-η_w ∇_Ŵ_i^(r)  (ℒ_T) 
    Probability update: α^(r)_i   α^(r)_i-η_α^(r)_i∇_α^(r)_i (ℒ_T)

Following parameter updates, the step size is reduced by a factor of 10 to define a more refined rank search space within the previous one, enabling fast convergence to the final search rank sets with a step size of one. Note that another way to reduce the computational cost of searching is by focusing on a smaller and sampled search space. This approach minimizes memory usage by considering only a subset of possible ranks during the search phase
<cit.>. However, our approach explores a broader search space, examining all possible ranks and allows to achieve an optimal compression rate. For each layer i, the tenor decomposition Ŵ_i^(r) of rank r corresponding to the highest probability p_i^(r) = softmax(α_i^(r)) enables reaching a smaller approximation error according to the total loss function (<ref>). For the next iteration, the selected rank SR_i for each layer i is the one that achieves the highest probability. The lower and upper bounds, Lb_i and Ub_i for defining the new rank search set are then set at a distance of half the step size around the selected rank SR_i.


Weight and probability updates, along with the search of the rank spaces, continue until the step size between the ranks of the final rank spaces is equal to one. 

ruled






 §.§ Final Decomposition and Fine-Tuning


The optimal selected ranks for the decomposition of tensor weights for each layer, denoted as ^⋆={SR_1,…,SR_n} are then determined from these final sets. With these optimal ranks, tensor weights are found by minimizing the decomposition loss:


    ℒ_d(W^^⋆) = ∑_i=1^nW_i-Ŵ_i^(SR_i)^2_F


To reduce the approximation error and  match closely the accuracy of the original network, we perform several epochs of feature fine-tuning through distillation using the training data D. In this approach, the original network serves as the teacher model and the decomposed model acts as the student network. The pseudocode for the overall procedure is presented in Algorithm <ref>.



 


§ EXPERIMENTS

We evaluated  on the CIFAR-10 <cit.> and ImageNet <cit.> datasets, testing it across various state-of-the-art deep neural networks. To demonstrate the generality of our approach, we applied both CP and TT decompositions. For TT decomposition, given the small filter dimensions in the convolutional layer, we performed decomposition in two dimensions and, due to computational resource constraints, assumed the two TT ranks to be equal. In the search phase, the initial lower and upperbounds for all layers were set to {10,100},{10,300}, {10,50} and {100,500} for ResNet-20, VGG-16, MobileNetV2 and ResNet-18, respectively, also for initial step size we set to {10, 10, 10, 100} for these models respectively. We used the standard SGD optimizer with Nesterov momentum set to 0.9, and hyperparameters α and β set to 0.2 and 0.6, respectively. The initial learning rates were 0.1 for CIFAR-10 and 0.01 for ImageNet, both scaled down using CosineAnnealingLR. For fine-tuning, we recovered accuracy over 10 epochs, using the same optimizer and learning rate schedule as in the search phase.



 §.§ Main Results


For the initial evaluation, we tested  on the CIFAR-10 dataset using ResNet-20 and VGG-16 models, with the results presented in Table <ref>.   yields competitive results compared to state-of-the-art (SOTA) methods. For ResNet-20,  with CP decomposition achieves 1.24% and 1.52% better performance in reduction of FlOPs and parameters, compated to the HALOC <cit.> method, while in accuracy, our approach with TT decomposition shows a 0.08% improvement. Additionally, with the VGG-16 model, we achieved significant compression rates and accuracy. Our approach with CP decomposition resulted in an 85.23% reduction in Flops, and a 98.6% reduction in parameters, while our approach with TT decomposition outperformed others in accuracy by 0.04%.



The results on the ImageNet dataset are presented in Table <ref>, where we evaluated   using ResNet-18 and MobileNetV2 models. For ResNet-18, our appraoch with CP decomposition yielded competitive results, while when using TT decomposition it outperformed the others, achieving state-of-the-art performance across all metrics, including Top-1, Top-5, reduction in FLOPs, and in parameters. In the case of MobileNetV2, we did not achieve high performance with the CP method. This is likely due to MobileNetV2's reliance on depthwise convolution, a low-rank convolution that does not benefit significantly from decomposition in certain dimensions. However,   with TT decomposition demonstrated superior compression results, achieving 1.86% and 2.31% greater reductions in FLOPs and parameters, respectively, along with competitive Top-1 and Top-5 accuracy. Our results underscore the importance of selecting the appropriate decomposition method based on the model's complexity. Our experiments indicate that TT decomposition is more effective for compressing higher-complexity models, such as those trained on the ImageNet dataset, while CP decomposition excels in compressing lower-complexity models, like those classically used on CIFAR-10.





 §.§ Analysis and Discussion

To gain deeper insights, we obtained additional results from the ResNet-18 experiment on the ImageNet dataset. Figure <ref> illustrates the ranks for both  CP and TT decompositions, highlighting that CP ranks are generally larger than those of TT due to the inherent differences in their decomposition methods. Additionally, Figure <ref> presents the behaviour of ℒ_T and ℒ_R during the two-step search process. The graph of ℒ_R shows greater fluctuations  during the first search phase compared to the second, as the variance between ranks decreases as the search progresses. Meanwhile, the graph of ℒ_T remains relatively stable, reflecting its composition of two distinct loss components. We also conducted an additional analysis on the effects of γ and β in the Eq <ref>, with the results provided in the supplementary materials.










 §.§ Searching Time

To compare the speed of   with HALOC, we measured the speed of ResNet-20 and VGG-16 on the CIFAR-10 dataset. In this experiment, we applied  CP decomposition for both methods, using the same rank search space and training each model for 100 epochs.   includes the time required for pretrained model as well as the search time, while HALOC involves training a SuperNet to find the optimal ranks. Table <ref> presents the search times for both methods.  demonstrated faster rank search in both models, resulting in a more efficient overall performance compared to HALOC. 







§ CONCLUSION


In this paper we presented an approach for deep neural-networks compression via decomposition and optimal rank selection. Our solution has two key features. It  takes into account all layers during the optimization process, aiming for high compression without sacrificing accuracy by identifying the optimal rank pattern across all layers. This approach leverages the observation that different layers contribute variably to the model's inference, allowing for smaller ranks in less critical layers and determining the most effective rank pattern for each. Furthermore, to achieve a high compression rate, we explore a broad range of ranks, addressing the significant memory challenges of this extensive exploration with a multistage rank search strategy. This strategy facilitates comprehensive exploration while ensuring efficient memory usage.







§ APPENDIX




 §.§ Rank Search Space function


The pseudocode of the the rank search space function is given in Algorithm <ref>. The function takes the lower and upper bounds for the ranks of each layer, denoted as Lb={Lb_1…,Lb_n} and Ub={Ub_1,…,Ub_n}, along with the step size, s, and the number of layers, n. For each layer, it then generates a list of ranks that define the search rank set. Each rank set ℛ_i contains m real values that are evenly spaced by the step s within the range from Lb_i to Ub_i. 

ruled





 §.§ Effect of γ and β on model performance

According to the definition of the total loss ℒ_T in Eq. (8) in the paper, higher values of  γ and β shift the balance between the decomposition loss and the rank constraint more towards the latter. This results in the optimization process selecting lower ranks, and vice versa. 
This trend is  illustrated in Figure <ref>, which displays the accuracy of the compressed model (a), the compression in terms of Flops (b) and the compression rate in terms of the percentage of fewer parameters for different values of γ and β for ResNet-20 using the CP decomposition. When β and γ are set to very high values, corresponding to larger selected ranks, the accuracy of the compressed model is low, while the compression in terms of Flops and the number of parameters is high. Conversely, when  γ and β are set to very low values, corresponding to lower selected ranks, these trends are reversed.






